Imaging device, solid-state imaging element, camera module, drive control unit, and imaging method

ABSTRACT

The present disclosure relates to an imaging device, a solid-state imaging element, a camera module, a drive control unit, and an imaging method for enabling reliable correction of an influence of a motion on an image. A state determination unit determines a state of a motion of an imaging unit that performs imaging to acquire an image via an optical system that collects light, and an exposure control unit performs at least control for an exposure time of the imaging unit according to a determination result by the state determination unit. Then, relative driving for the optical system or the imaging unit is performed to optically correct a blur appearing in the image according to an exposure period of one frame by the exposure time, and driving for resetting a relative positional relationship between the optical system and the imaging unit, the relative positional relationship being caused during the exposure period, is performed according to a non-exposure period in which exposure is not performed between the frames. The present technology can be applied to, for example, a stacked CMOS image sensor.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Phase of International PatentApplication No. PCT/JP2019/001763 filed on Jan. 22, 2019, which claimspriority benefit of Japanese Patent Application No. JP 2018-018481 filedin the Japan Patent Office on Feb. 5, 2018. Each of the above-referencedapplications is hereby incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates to an imaging device, a solid-stateimaging element, a camera module, a drive control unit, and an imagingmethod, and particularly to an imaging device, a solid-state imagingelement, a camera module, a drive control unit, and an imaging methodfor enabling reliable correction of an influence of a motion of animaging unit on an image.

BACKGROUND ART

Conventionally, as a technology for correcting camera shake in animaging device, optical image stabilizer (OIS) or electronic imagestabilization (EIS) has been used. In the optical image stabilizer, itis possible to correct a blur by relatively moving a lens or an imagingelement in parallel according to the amount of blur and shifting aposition of an image on the imaging element. In the electronic imagestabilization, it is possible to correct a blur by cutting out an imagecaptured by an imaging element and adopting the cutout image as anoutput image, and shifting a cutout position according to the amount ofblur.

By the way, a main cause of actual camera shake is a rotational motion,and the influence of a parallel motion is small, and in particular, theinfluence of a parallel motion becomes smaller as a distance to anobject becomes more distant. In the optical image stabilizer technology,there are some cases where a periphery is deformed because therotational motion is corrected according to the parallel motion of alens or an imaging element. Similarly, the electronic imagestabilization has the problem that a periphery is deformed because thecorrection is conducted by moving the cutout position in parallel.

Furthermore, measures against deformation (focal plane phenomenon) hasnot been taken, the deformation being caused by a difference in a movingamount in one screen due to deviation of an exposure time for each lineof a pixel, the deviation occurring in an imaging element using arolling shutter such as a complementary metal oxide semiconductor (CMOS)image sensor.

Therefore, as disclosed in Patent Document 1, an imaging device capableof performing image stabilization corresponding to a difference in amoving amount depending on a position in an image plane or a differencein a moving amount due to deviation of an exposure time in one screenhas been proposed. By adopting the image stabilization, the camera shakecan be corrected from the center to the periphery with extremely highaccuracy, and the deformation due to the focal plane phenomenon can alsobe corrected.

CITATION LIST Patent Document

-   Patent Document 1: International Publication No. 2014/156731

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

By the way, in a case where imaging is performed outdoors during thedaytime or the like with a short exposure time, the above-describedimage stabilization in Patent Document 1 can almost completely correctthe influence of camera shake and suppress occurrence of a blur anddeformation in an image. However, in a case where imaging is performedin a dark place, during the night time, or the like with a long exposuretime, the above-described image stabilization in Patent Document 1 cancorrect occurrence of positional deviation and deformation in an imagebut has a difficulty in suppressing occurrence of a blur (hereinafterreferred to as a blur during an exposure time or an exposure blur) of apoint image during exposure.

The present disclosure has been made in view of such a situation, and tosuppress occurrence of a blur during an exposure time and enablereliable correction of an influence of a motion of an imaging unit on animage.

Solutions to Problems

An imaging device according to one aspect of the present disclosureincludes a state determination unit configured to determine a state of amotion of an imaging unit that performs imaging to acquire an image viaan optical system that collects light, and an exposure control unitconfigured to perform at least control for an exposure time of theimaging unit according to a determination result by the statedetermination unit, in which relative driving for the optical system orthe imaging unit is performed to optically correct a blur appearing inthe image according to an exposure period of one frame by the exposuretime, and driving for resetting a relative positional relationshipbetween the optical system and the imaging unit, the relative positionalrelationship being caused during the exposure period, is performedaccording to a non-exposure period in which exposure is not performedbetween the frames.

A solid-state imaging element according to one aspect of the presentdisclosure is configured by stacking a semiconductor chip on which animaging unit that performs imaging to acquire an image via an opticalsystem that collects light is formed, and a semiconductor chip on whicha logic unit including a state determination unit that determines astate of a motion of the imaging unit and an exposure control unit thatperforms at least control for an exposure time of the imaging unitaccording to a determination result by the state determination unit isformed, in which relative driving for the optical system or the imagingunit is performed to optically correct a blur appearing in the imageaccording to an exposure period of one frame by the exposure time, anddriving for resetting a relative positional relationship between theoptical system and the imaging unit, the relative positionalrelationship being caused during the exposure period, is performedaccording to a non-exposure period in which exposure is not performedbetween the frames.

A camera module according to one aspect of the present disclosureincludes an optical system configured to collect light, an imaging unitconfigured to perform imaging via the optical system to acquire animage, a state determination unit configured to determine a state of amotion of the imaging unit, and an exposure control unit configured toperform at least control for an exposure time of the imaging unitaccording to a determination result by the state determination unit, inwhich relative driving for the optical system or the imaging unit isperformed to optically correct a blur appearing in the image accordingto an exposure period of one frame by the exposure time, and driving forresetting a relative positional relationship between the optical systemand the imaging unit, the relative positional relationship being causedduring the exposure period, is performed according to a non-exposureperiod in which exposure is not performed between the frames.

A drive control unit according to the one aspect of the presentdisclosure in which at least control for an exposure time of an imagingunit that performs imaging to acquire an image via an optical systemthat collects light is performed according to a determination result ofa state of a motion of the imaging unit,

the drive control unit configured to

control relative driving for the optical system or the imaging unit tooptically correct a blur appearing in the image according to an exposureperiod of one frame by the exposure time; and

control driving for resetting a relative positional relationship betweenthe optical system and the imaging unit, the relative positionalrelationship being caused during the exposure period, according to anon-exposure period in which exposure is not performed between theframes.

An imaging method according to one aspect of the present disclosureincludes, by an imaging device including an imaging unit that performsimaging to acquire an image via an optical system that collects light,determining a state of a motion of the imaging unit, and performing atleast control for an exposure time of the imaging unit according to adetermination result by the state determination unit, in which relativedriving for the optical system or the imaging unit is performed tooptically correct a blur appearing in the image according to an exposureperiod of one frame by the exposure time, and driving for resetting arelative positional relationship between the optical system and theimaging unit, the relative positional relationship being caused duringthe exposure period, is performed according to a non-exposure period inwhich exposure is not performed between the frames.

According to one aspect of the present disclosure, a state of a motionof an imaging unit that performs imaging to acquire an image via anoptical system that collects light is determined, at least control foran exposure time of the imaging unit is performed according to thedetermination result, relative driving for the optical system or theimaging unit is performed to optically correct a blur appearing in theimage according to an exposure period of one frame by the exposure time,and driving for resetting a relative positional relationship between theoptical system and the imaging unit, the relative positionalrelationship being caused during the exposure period, is performedaccording to a non-exposure period in which exposure is not performedbetween the frames.

Effects of the Invention

According to one aspect of the present disclosure, the influence of amotion of an imaging unit on an image can be reliably corrected.

BRIEF DESCRIPTION OF DRAWINGS

FIGS. 1A, 1B, and 1C are diagrams illustrating examples of distortiongenerated in an image due to an influence of lens distortion.

FIG. 2 is a diagram illustrating vibration conditions applied to animaging device.

FIG. 3 is a diagram illustrating examples of images output withoutcorrecting a blur.

FIG. 4 is a diagram illustrating examples of images to which correctionprocessing by ordinary electronic image stabilization has been applied.

FIG. 5 is a diagram illustrating examples of images to which correctionprocessing by optical image stabilizer has been applied.

FIG. 6 is a diagram illustrating examples of images to which correctionprocessing by image stabilization proposed in Patent Document 1 has beenapplied.

FIG. 7 is a diagram illustrating examples of images to which correctionprocessing by image stabilization proposed in Patent Document 2 has beenapplied but correction for lens distortion itself is not performed.

FIG. 8 is a diagram illustrating examples of images to which correctionprocessing by image stabilization proposed in Patent Document 2 has beenapplied and correction for lens distortion itself is performed.

FIG. 9 is a diagram illustrating examples of images to be output withoutcorrecting a blur in a case where imaging is performed with a longexposure time.

FIG. 10 is a diagram illustrating examples of images to which correctionprocessing by image stabilization proposed in Patent Document 2 has beenapplied and correction for lens distortion is performed in a case whereimaging is performed with a long exposure time.

FIG. 11 is a diagram illustrating examples of images to which correctionprocessing by optical image stabilizer has been applied in a case whereimaging is performed with a long exposure time.

FIG. 12 is a diagram illustrating examples of images to which correctionprocessing by image stabilization used in the present technology hasbeen applied.

FIG. 13 is a block diagram illustrating a configuration example of animaging device for describing image stabilization used in the presenttechnology.

FIG. 14 is a graph for describing correction processing by electronicimage stabilization by a signal processing circuit.

FIG. 15 is a flowchart for describing image stabilization processingexecuted in an imaging method by the imaging device.

FIG. 16 is a graph for describing image correction results.

FIG. 17 is a block diagram illustrating a configuration example of anembodiment of an imaging device to which the present technology isapplied.

FIG. 18 is a diagram for describing exposure control and OIS controlinformation in a first vibration state.

FIG. 19 is a diagram for describing vibration that exceeds a correctablerange when the exposure control in the first vibration state is beingperformed.

FIG. 20 is a diagram for describing exposure control and OIS controlinformation in a second vibration state.

FIG. 21 is a diagram for describing vibration that exceeds a correctablerange when the exposure control in the second vibration state is beingperformed.

FIG. 22 is a diagram for describing exposure control and OIS controlinformation in a third vibration state.

FIG. 23 is a diagram for describing an example of transition of avibration state in a first exposure time control mode.

FIG. 24 is a flowchart for describing image stabilization processing inthe first exposure time control mode.

FIG. 25 is a diagram for describing an example of transition of avibration state in a second exposure time control mode.

FIG. 26 is a flowchart for describing image stabilization processing inthe second exposure time control mode.

FIG. 27 is a diagram for describing a modification of transition of thevibration state in the second exposure time control mode.

FIG. 28 is a diagram illustrating definitions of a pitch direction, ayaw direction, and a roll direction.

FIG. 29 is a diagram illustrating use examples of an image sensor.

MODE FOR CARRYING OUT THE INVENTION

First, vibration and image stabilization processing of an imaging devicewill be described with reference to FIGS. 1A, 1B, 1C, 2, 3, 4, 5, 6, 7,8, 9, 10, 11, and 12 before describing the imaging device to which thepresent technology is applied.

<Vibration and Image Stabilization Processing of Imaging Device>

FIGS. 1A, 1B, and 1C illustrate examples of distortion generated inimages due to an influence of lens distortion when an object is capturedby an imaging device.

For example, when a lattice pattern as illustrated in FIG. 1A iscaptured as an object, pincushion distortion where a peripheral portionshrinks inward occurs as illustrated in FIG. 1B or barrel distortionwhere the peripheral portion extends outward occurs as illustrated inFIG. 1C.

Hereinafter, description will be given regarding a blur that isgenerated in an image when imaging is performed while providingvibration under vibration conditions (an oscillation angle in a yawdirection: 1.5 degrees and an oscillation angle in a pitch direction:1.2 degrees) illustrated in FIG. 2 in an imaging device having apincushion lens distortion as illustrated in FIG. 1B. The left side inFIGS. 3 to 12 illustrates four images captured at around points wherethe oscillation angle becomes minimum (for example, two points where theoscillation angle becomes 0 degrees) and around points where theoscillation angle becomes maximum (for example, two points where the yawoscillation angle becomes 1.5 degrees and −1.5 degrees) in one cycle ofvibration illustrated in FIG. 2. Furthermore, the right side in FIGS. 3to 12 illustrates an image in which the four images are superimposed.

FIG. 3 illustrates examples of images to be output without applyingcorrection processing for correcting a blur to the vibration.

As illustrated in FIG. 3, various deformations according to positions inan image plane are generated due to influences of positional deviation,peripheral deformation, and a rolling shutter caused by the vibration ofthe imaging device.

FIG. 4 illustrates examples of images to which correction processing byordinary electronic image stabilization has been applied. Here, theordinary electronic image stabilization is to correct a blur by cuttingout an image captured by an imaging element and adopting the cutoutimage as an output image, and shifting a cutout position according to ablur amount, which is different from the above-described correctionprocessing in Patent Document 1 and correction processing in PatentDocument 2 to be described below.

As illustrated on the left side in FIG. 4, by applying the correctionprocessing by the normal electronic image stabilization, images with ashifted cutout position according to a blur amount are output.Therefore, when these images are superimposed, the images are correctedsuch that center positions on the screen of the images between framescoincide with one another, as illustrated on the right side in FIG. 4.However, the correction processing by the ordinary electronic imagestabilization cannot correct the peripheral deformation due to therolling shutter or camera shake.

FIG. 5 illustrates examples of images to which correction processing byoptical image stabilizer has been applied.

As illustrated on the left side in FIG. 5, by applying the correctionprocessing by the optical image stabilizer, images captured whilerelatively moving the lens or the imaging element in parallel accordingto the blur amount are output. Therefore, when these images aresuperimposed, center positions on the screen of the images betweenframes can be corrected and the influence of the rolling shutter canalso be corrected, as illustrated on the right side in FIG. 5. However,in this case, the peripheral deformation due to camera shake cannot becorrected.

Note that, in the optical image stabilizer (a barrel shift method or asensor shift method), occurrence of a blur during an exposure time canbe suppressed by applying the correction processing to follow thevibration during exposure although the influence of the peripheraldeformation and the lens distortion remains.

FIG. 6 illustrates examples of images to which correction processing byimage stabilization proposed in above-described Patent Document 1 hasbeen applied. In the image stabilization in Patent Document 1, not onlythe cutout position is shifted like the ordinary electronic imagestabilization but also deformation is performed for each pixel positioncorresponding to the difference in a moving amount depending on aposition in an image plane or the difference in a moving amount due toexposure time deviation in one screen.

As illustrated in FIG. 6, by applying the correction processing by theimage stabilization proposed in Patent Document 1, the blur is reliablycorrected from the center to the periphery of the images. Note that,since the influence of lens distortion is not taken into considerationin this correction processing, there is an error due to the influence oflens distortion in an actual capture result, and some positionaldeviation occurs in a peripheral portion. Note that this positionaldeviation varies depending on the shape of the lens distortion.

Therefore, as previously filed in PCT/JP2016/070261 (hereinafterreferred to Patent Document 2), correction processing capable of imagestabilization in consideration of the influence of lens distortion hasbeen proposed.

Images to which correction processing by the image stabilizationproposed in Patent Document 2 will be described with reference to FIGS.7 and 8.

FIG. 7 illustrates examples of images to which the correction processingby the image stabilization in Patent Document 2 has been applied, and towhich correction for the lens distortion itself is not performed whilecorrection for deformation due to camera shake generated by theinfluence of the lens distortion is performed.

As illustrated in FIG. 7, by applying the correction processing by theimage stabilization proposed in Patent Document 2, the blur is reliablycorrected from the center to the periphery of the images.

FIG. 8 illustrates examples of images to which the correction processingby the image stabilization in Patent Document 2 has been applied, and towhich correction for deformation due to camera shake generated by theinfluence of the lens distortion is performed and correction for thelens distortion itself is performed.

As illustrated in FIG. 8, by applying the correction processing by theimage stabilization proposed in Patent Document 2, the blur is reliablycorrected from the center to the periphery of the images after the lensdistortion is corrected.

By the way, in a case where imaging is performed outdoors during thedaytime or the like with a short exposure time, the image stabilizationcan be almost completely performed by the correction processing by theimage stabilization proposed in Patent Document 2. In contrast, in acase where imaging is performed in a dark place, during the night time,or the like with a long exposure time, a blur during the exposure timeoccurs.

FIG. 9 illustrates images to be output without correcting a blur due tothe vibration, similarly to FIG. 3, in the case where imaging isperformed with a long exposure time.

As illustrated in FIG. 9, various deformations according to positions inan image plane are generated due to the influences of positionaldeviation, peripheral deformation, and the rolling shutter caused by theimaging with a long exposure time. In addition, a blur during theexposure time occurs.

FIG. 10 illustrates images to which the correction processing by theimage stabilization in Patent Document 2 has been applied, and to whichcorrection for deformation due to camera shake generated by theinfluence of the lens distortion is performed and correction for thelens distortion itself is performed, in the case where imaging isperformed with a long exposure time.

Even by the correction processing by the image stabilization proposed inPatent Document 2, the positions coincide in the corrected images, butthe images become images in which the camera shake has not been reliablycorrected due to occurrence of the blur during the exposure time. Thatis, the images have the blur during the exposure time while thepositional deviation, deformation, or the like due to the camera shakehas been corrected from the center to the periphery after the lensdistortion is corrected.

Furthermore, FIG. 11 illustrates images to which the correctionprocessing by the optical image stabilizer (barrel shift method orsensor shift method) has been applied in the case where imaging isperformed with a long exposure time.

As illustrated in FIG. 11, in the correction processing by the opticalimage stabilizer, the peripheral deformation due to camera shake cannotbe corrected. However, occurrence of the blur during the exposure timecan be suppressed by relative movement of the lens and the imagingelement during the exposure time.

Therefore, the applicant of the present application proposes correctionprocessing for suppressing the influence provided by camera shake on animage and reliably correcting a blur in the image even in a case wherethe exposure time is long, as in an imaging device 11 in FIG. 13 or animaging device 11A in FIG. 17 to be described below. By the correctionprocessing, the blur can be corrected from the center to the peripheryafter the lens distortion is corrected, and occurrence of the blurduring the exposure time can be suppressed.

That is, as illustrated in FIG. 12, by applying the correctionprocessing to which the present technology is applied, the blur can bereliably corrected from the center to the periphery, and occurrence ofthe blur during the exposure time can be suppressed, after the lensdistortion is corrected. Note that, since an exposure blur due toperipheral deformation during the exposure time cannot be suppressed bythe optical image stabilizer, it is assumed that some blur during theexposure time remains in the periphery of the image depending on thevibration conditions and the exposure time. However, the blur during theexposure time can be suppressed to the extent that the blur is hardlynoticeable in normal capture.

Furthermore, FIG. 12 illustrates images for which correction for thelens distortion itself has been performed, similarly to FIG. 8. Notethat, although not illustrated, even in a case where correction for thelens distortion itself is not performed, the blur can be reliablycorrected from the center to the periphery and occurrence of the blurduring the exposure time can be suppressed, similarly to FIG. 7.

<Image Stabilization Used in Present Technology>

Hereinafter, specific embodiments to which the present technology isapplied will be described in detail with reference to the drawings.First, the image stabilization used in the present technology will bedescribed with reference to FIGS. 13 to 16.

FIG. 13 is a block diagram illustrating a configuration example of animaging device for describing image stabilization used in the presenttechnology.

As illustrated in FIG. 13, the imaging device 11 includes a lens unit12, an image sensor 13, a motion sensor 14, an OIS driver 15, an OISactuator 16, a signal processing circuit 17, a display 18, and arecording medium 19.

The lens unit 12 includes one or a plurality of lenses, collects lightfrom an object, and forms an image of the object on a sensor surface ofan imaging unit 21 included in the image sensor 13.

The image sensor 13 is configured by stacking a semiconductor chip onwhich the imaging unit 21 is formed and a semiconductor chip on which alogic unit 22 is formed, and an interface for taking an output from theOIS driver 15 is mounted.

The imaging unit 21 has light collected from the object by the lens unit12, captures an image of the object formed on the sensor surface inwhich a plurality of pixels is arranged in a matrix, and outputs animage acquired by the capturing.

The logic unit 22 supplies, to the signal processing circuit 17, imagedata in which positional information of the lens unit 12 and angularvelocity data output from the OIS driver 15 are added together withtiming information indicating timing to synchronize the aforementioneddata with coordinates on the image to the image captured by the imagingunit 21.

Specifically, the logic unit 22 receives the angular velocity datadetected by the motion sensor 14 and the positional information of thelens unit 12 driven by the OIS actuator 16 at a predetermined samplingfrequency (for example, 1 kHz) from the OIS driver 15. Then, the logicunit 22 adds the positional information of the lens unit 12 and theangular velocity data, and an H line counter of the image data at thetiming when the aforementioned data are received, to the image data, andoutputs the image data. Of course, the positional information of thelens unit 12, the angular velocity data, and the H line counter may beindividually output together with the image data without being added tothe image data. In this way, the positional information of the lens unit12 and the angular velocity data are associated in units of a horizontalline of the image data, so that the angular velocity data and thepositional information, and a vertical position of the image can besynchronized in the signal processing circuit 17. That is, the H linecounter is used as the timing information for synchronizing the data.

Here, for example, the H line counter of the image data is a counterthat is reset for each frame at predetermined timing and incrementsevery time a horizontal one line is read, and is used for timing thevertical position of the image. Note that the H line counter is countedin a blank section in which no image is read. Furthermore, as the timinginformation, time information such as a time stamp may be used, forexample, other than use of the H line counter of the image data. Notethat the method of synchronizing the angular velocity data and thepositional information, and the vertical position of the image isdescribed in detail in Patent Document 2 above.

The motion sensor 14 physically (not by image processing) detects amotion of the imaging unit 21 and outputs information indicating themotion. For example, in a case where the motion sensor 14 is configuredby a gyro sensor capable of detecting angular velocities in three-axialdirections, the motion sensor 14 outputs angular velocity datarepresented by the angular velocities as the information indicating themotion of the imaging device 11.

Note that, as the motion sensor 14, a single gyro sensor, a gyro sensorshared for OIS control (that is, having two ports), or the like can beused, for example, other than use of the gyro sensor for OIS control.Furthermore, the motion sensor 14 is not limited to a gyro sensor, and asix-axis sensor capable of outputting acceleration data and the like inaddition to the angular velocity data in the three-axis directions canbe used.

The OIS driver 15 calculates a moving amount for moving the lens unit 12so as to optically cancel occurrence of a blur in the image captured bythe imaging unit 21, on the basis of the angular velocity data outputfrom the motion sensor 14. Then, the OIS driver 15 supplies thecalculated moving amount to the OIS actuator 16, and performs control toarrange the lens unit 12 at a predetermined position according to themoving amount. Moreover, the OIS driver 15 acquires the positionalinformation of the lens unit 12 driven by the OIS actuator 16, andoutputs the positional information of the lens unit 12 and the angularvelocity data to the image sensor 13.

The OIS actuator 16 drives the lens unit 12 according to the movingamount supplied from the OIS driver 15, thereby optically correctingcamera shake occurring in the image captured by the image sensor 13.Then, the OIS actuator 16 detects the position of the lens unit 12according to the driving, and supplies the positional information of thelens unit 12 to the OIS driver 15.

The signal processing circuit 17 is configured to perform correctionprocessing similar to the electronic image stabilization proposed inPatent Document 2 while taking the positional information of the lensunit 12 into consideration. That is, the signal processing circuit 17applies signal processing for correcting the influence of the motion ofthe imaging unit 21 on the image (for example, the positional deviation,peripheral deformation, distortion by the rolling shutter, deformationby the influence of the lens distortion, and the like) according to afunction for performing correction on the basis of the positionalinformation of the lens unit 12 and the angular velocity datasynchronized for each coordinate on the image, on the basis of thepositional information of the lens unit 12 and the angular velocity dataadded to the image data supplied from the image sensor 13. Note that thecorrection processing by the signal processing circuit 17 will bedescribed below with reference to FIG. 14.

The display 18 includes a display unit such as a liquid crystal panel oran organic electro luminescence (EL) panel, for example, and displays animage output by the signal processing circuit 17.

The recording medium 19 is a memory built in the imaging device 11 or amemory removably attached to the imaging device (the memory is anelectronically erasable and programmable read only memory (EEPROM), forexample), and records the image output by the signal processing circuit17.

The imaging device 11 is thus configured, and the signal processingcircuit 17 can apply the correction processing by the electronic imagestabilization to the image captured by the image sensor 13 such thatoccurrence of a blur is optically suppressed. Thereby, the imagingdevice 11 can suppress occurrence of the blur during the exposure timeand reliably correct the blurs of the image (such as the positionaldeviation caused by camera shake, peripheral deformation, distortion bythe rolling shutter, and deformation by the influence of lensdistortion), as illustrated in FIG. 12 above.

Note that, in the present embodiment, the barrel shift optical imagestabilizer in which the lens unit 12 is driven by the OIS actuator 16 isdescribed. However, the imaging device 11 may adopt sensor shift opticalimage stabilizer in which the image sensor 13 is driven by the OISactuator 16. In this case, the OIS actuator 16 supplies the positionalinformation of the image sensor 13, instead of the positionalinformation of the lens unit 12, to the OIS driver 15.

Furthermore, in the imaging device 11 in FIG. 13, the angular velocitydata output from the motion sensor 14 is supplied to the image sensor 13via the OIS driver 15. In contrast, the imaging device 11 may have aconfiguration in which the motion sensor 14 has two output ports usedfor outputting the angular velocity data, and the motion sensor 14supplies the angular velocity data to the image sensor 13 and the OISdriver 15, for example. In this case, the angular velocity data is notsupplied from the OIS driver 15 to the image sensor 13.

Alternatively, the imaging device 11 may include two motion sensors 14,for example. In this case, the two motion sensors 14 supply the angularvelocity data to the image sensor 13 and the OIS driver 15. Also in thiscase, the angular velocity data is not supplied from the OIS driver 15to the image sensor 13.

Moreover, in the imaging device 11 in FIG. 13, the image sensor 13 andthe signal processing circuit 17 are illustrated as different blocks.However, a configuration in which the processing by the signalprocessing circuit 17 is performed inside the image sensor 13 may beadopted, for example. That is, the image sensor 13 can have a stackedstructure in which the semiconductor chip on which the signal processingcircuit 17 is formed is stacked.

The correction processing by the electronic image stabilization by thesignal processing circuit 17 will be described with reference to FIG.14.

As illustrated in FIG. 14, an optical center of an output image by theimage sensor 13 is a position O (0, 0). Then, it is assumed that theimage sensor 13 rotates at a rotation angle −θp (rad) in the pitchdirection, at a rotation angle −θy (rad) in the yaw direction, and at arotation angle −θr (rad) in the roll direction. It is assumed that apoint p (x, y) captured on the image sensor 13 moves to a point P (X, Y)by such rotation.

Moreover, the image stabilization relational expression disclosed inPatent Document 1 above, that is, the following expressions (1) areestablished, where a point p0 (x0, y0) is coordinates of the point p (x,y) with corrected distortion, and a point P0 (X0, Y0) is coordinates ofthe point P (X, Y) with corrected distortion.

$\begin{matrix}\lbrack {{Math}.\mspace{14mu} 1} \rbrack & \; \\\{ {{\begin{matrix}\begin{matrix}{{X\; 0} = {{L \cdot ( {\tan( {\alpha + \theta_{y}} )} )} + {x\;{0 \cdot \frac{\;{\cos\;\beta}}{\cos( {\beta + \theta_{p}} )}}} +}} \\{{x\;{0 \cdot \cos}\theta_{r}} - {y\;{0 \cdot \sin}\theta_{r}} - {{2 \cdot x}\; 0}}\end{matrix} \\\begin{matrix}{{Y\; 0} = {{L \cdot ( {\tan( {\beta + \theta_{p}} )} )} + {y\;{0 \cdot \frac{\cos\;\alpha}{\cos( {\alpha + \theta_{y}} )}}} +}} \\{{x\;{0 \cdot \sin}\theta_{r}} + {y\;{0 \cdot \cos}\theta_{r}} - {{2 \cdot y}\; 0}}\end{matrix}\end{matrix}\tan\alpha} = {{\frac{x0}{L}\tan\;\beta} = \frac{y0}{L}}}  & (1)\end{matrix}$

Note that, in the expressions (1), a focal length L is a focal length atan optical center position of the image sensor 13, which is convertedinto the number of pixels, and has a value that satisfies the followingexpression (2), using a moving amount d of the position O (0, 0) of theoptical center when the image sensor 13 rotates at a rotation angle −θin the pitch direction or the yaw direction.[Math. 2]d=L·tan θ  (2)

Furthermore, the P0(X0, Y0) is expressed by the following expression(3), where the image stabilization relational expression of theabove-described expression (1), that is, calculation of obtaining thepoint P0 (X0, Y0) from the point p0 (x0, y0) is a function T.[Math. 3]P0(X0,Y0)=T(x0,y0,L,θ _(p),θ_(y),θ_(r))

Moreover, the point p0 (x0, y0) is expressed by the following expression(4), where calculation of obtaining the point p0 (x0, y0) from the pointp (x, y), that is, calculation of obtaining where a point on the imageaffected by the lens distortion is located in a case where there is nolens distortion is a function U.[Math. 4]p0(x0,y0)=U(x,y)  (4)

Furthermore, the point P (X, Y) is expressed by the following expression(5), where calculation of obtaining the point P (X, Y) from the point P0(X0, Y0), that is, calculation of obtaining where on the image affectedby the lens distortion a point on the image in the case where there isno lens distortion is located is a function D.[Math. 5]P(X,Y)=D(X0,Y0)  (5)

Then, for example, in the case of outputting a result with correctedlens distortion as illustrated in FIG. 8 by the correction processing bythe signal processing circuit 17, a point on an output image may beregarded as the point p0 (x0, y0). That is, by using a pixel value ofthe point P (X, Y) as a pixel value of the point p0 (x0, y0), for eachpoint in the output image, an image to which the image stabilization isapplied and the lens distortion is corrected can be obtained.

At this time, the point P (X, Y) can be obtained from the point p0 (x0,y0), using the function T of the expression (3) and the function D ofthe expression (5). That is, the point P0 (X0, Y0) can be obtained fromthe point p0 (x0, y0), using the function T of the expression (3).Moreover, the point P (X, Y) can be obtained from the point P0 (X0, Y0),using the function D of the expression (5). Here, the point P (X, Y) isexpressed by the following expression (6), where a composite function ofthe function T and the function D is a function F.[Math. 6]P(X,Y)=F(x0,y0,L,θ _(p),θ_(y),θ_(r))  (6)

Meanwhile, for example, in the case of outputting a result withoutcorrected lens distortion as illustrated in FIG. 7 by the correctionprocessing by the signal processing circuit 17, a point on an outputimage may be regarded as the point p (x, y). That is, by using the pixelvalue of the point P (X, Y) as the pixel value of the point p (x, y),for each point in the output image, an image to which the imagestabilization is applied and the lens distortion is not corrected can beobtained.

At this time, the point P (X, Y) can be obtained from the point p (x,y), using the function T of the expression (3), the function U of theexpression (4), and the function D of the expression (5). That is, thepoint p0 (x0, y0) can be obtained from the point p (x, y), using thefunction U of the expression (4). The point P0 (X0, Y0) can be obtainedfrom the point p0 (x0, y0), using the function T of the expression (3).Moreover, the point P (X, Y) can be obtained from the point P0 (X0, Y0),using the function D of the expression (5). Here, the point P (X, Y) isexpressed by the following expression (7), where a composite function ofthe function T, the function U, and the function D is a function G.[Math. 7]P(X,Y)=G(x,y,L,θ _(p),θ_(y),θ_(r))  (7)

Note that the coordinate value of the point P (X, Y) obtained by theexpressions (6) or (7) rarely becomes an integer value, but the pixelvalue of the output image can be calculated from pixel values of thenearby coordinates by interpolation. Furthermore, to obtain the pixelvalue of each point of the output image, the pixel value can becalculated by calculating a corresponding coordinate position on aninput image using the above-described functions for each point.Alternatively, for example, the pixel value may be calculated bydividing the output image, calculating corresponding coordinatepositions on the input image for only grid points using theabove-described functions, and obtaining coordinate positions for pointsother than the grid points by interpolation calculation.

Note that, here, to simply describe the principle, calculation of thepixel value at certain timing has been described. However, in reality,the capture time of a pixel in one screen is different depending on thepixel position. Therefore, the pixel value is calculated at each pixelusing the pitch rotation angle −θp (rad), yaw rotation angle −θy (rad),and roll rotation angle −θr (rad) according to the pixel position.

By the way, by adding the moving amount for moving the lens unit 12 bythe OIS actuator 16 to the correction processing by the function F ofthe expression (6) and the function G of the expression (7) above, thecorrection processing by the optical image stabilizer and the electronicimage stabilization is implemented Hereinafter, the correctionprocessing using the function F of the expression (6) will be described.However, the correction processing using the function G of theexpression (7) can also be implemented similarly to the correctionprocessing using the function F of the expression (6)

First, assuming that coordinates on the input image (camera shake imagein the case of no optical image stabilizer) corresponding to thecoordinates of the point p0 (x0, y0) on the output image to which thecorrection processing by the optical image stabilizer and the electronicimage stabilization is applied is the point P0 (X0, Y0). At this time,as described above, the function F for calculating the coordinates bythe electronic image stabilization is expressed by the expression (6).

Furthermore, the correction processing by the optical image stabilizer(barrel shift method or sensor shift method) may be considered asparallel movement of the image. Coordinates (X_(ois), Y_(ois)) on theinput image corresponding to the coordinates p0 (x0, y0) on the outputimage are obtained by the following expression (8), using a shift amount(x_(ois), y_(ois)) by the optical image stabilizer for each pixel, onthe basis of the above assumption.

$\begin{matrix}\lbrack {{Math}.\mspace{14mu} 8} \rbrack & \; \\\begin{matrix}{( {X_{ois},Y_{ois}} ) = {( {X,Y} ) - ( {x_{ois},y_{ois}} )}} \\{= {{F( {{x\; 0},{y\; 0},L,\theta_{p},\theta_{y},\theta_{r}} )} - ( {x_{ois},y_{ois}} )}}\end{matrix} & (8)\end{matrix}$

Therefore, by outputting the pixel value of the coordinates (X_(ois),Y_(ois)) on the input image as the pixel value of the coordinates (x0,y0) on the output image, an image to which the correction processing bythe optical image stabilizer and the electronic image stabilization isapplied can be output.

Note that the coordinate value of the coordinates (X_(ois), Y_(ois))obtained by the expression (8) rarely becomes an integer value, but thepixel value of the output image can be calculated from pixel values ofthe nearby coordinates by interpolation. Furthermore, to obtain thepixel value of each point of the output image, the pixel value can becalculated by calculating a corresponding coordinate position on aninput image using the above-described functions for each point.Alternatively, for example, the pixel value may be calculated bydividing the output image, calculating corresponding coordinatepositions on the input image for only grid points using theabove-described functions, and obtaining coordinate positions for pointsother than the grid points by interpolation calculation.

Note that, here, to simply describe the principle, calculation of thepixel value at certain timing has been described. However, in reality,the capture time of a pixel in one screen is different depending on thepixel position. Therefore, the pixel value is calculated at each pixelusing the pitch rotation angle −θp (rad), yaw rotation angle −θy (rad),roll rotation angle −θr (rad) according to the pixel position, and theshift amount by the optical image stabilizer.

Here, a case of using Hall data obtained by reading the position of thelens unit 12 using a Hall element as the positional information of thelens unit 12 driven in the optical image stabilizer will be described.For example, the logic unit 22 can add the angular velocity datadetected by the motion sensor 14 and the Hall data obtained by readingthe position of the lens unit 12 together with the H line counter of theimage data to the image data, and output the image data. At this time,the logic unit 22 adjusts timing to synchronize delay time to timingwhen the motion sensor 14 has detected the angular velocity data and totiming when the Hall element has read the position of the lens unit 12,an exposure end time, and an exposure time for each pixel (H line), withthe acquisition timing of the angular velocity data and the Hall data,on the basis of the relationship among the aforementioned times and thelike.

In this case, the coordinates (x0, y0) on the output image (image afterthe image stabilization) are obtained by the following expression (9),using the shift amount (x_(ois), y_(ois)) by the optical imagestabilizer, a Hall data value (hx, hy), Hall data (hx0, hy0) of when thelens unit 12 is located in the center by the optical image stabilizer,and a pixel number conversion coefficient (kx, ky).[Math. 9](x _(ois) ,y _(ois))=(kx·(hx−hx0),ky·(hy−hy0))  (9)

Then, by inputting the shift amount (x_(ois), y_(ois)) obtained by theexpression (9) to the above expression (8), the coordinates (X_(ois),Y_(ois)) on the input image (OIS output image) corresponding to thecoordinates p0 (x0, y0) on the output image (image after the imagestabilization) are determined. Thereby, an image stabilization image canbe created by using the pixel value of the coordinates. Here, theexpression (9) indicates an example of a case where conversionprocessing is performed assuming that a change amount of the Hall data(hx0, hy0) and the moving amount of the pixel position have a linearrelationship. In contrast, in a case where there is no linearrelationship, for example, conversion processing according to therelationship between the change amount of the Hall data (hx0, hy0) andthe moving amount of the pixel position is performed.

Note that, to obtain the pixel value of each point of the output image,the pixel value can be calculated by calculating a correspondingcoordinate position on an input image using the above-described functionfor each point. Alternatively, for example, the pixel value may becalculated by dividing the output image, calculating correspondingcoordinate positions on the input image for only grid points using theabove-described functions, and obtaining coordinate positions for pointsother than the grid points by interpolation calculation.

Note that, here, to simply describe the principle, calculation of thepixel value at certain timing has been described. However, in reality,the capture time of a pixel in one screen is different depending on thepixel position.

Therefore, the pixel value is calculated at each pixel using the pitchrotation angle −θp (rad), yaw rotation angle −θy (rad), roll rotationangle −θr (rad) according to the pixel position, and the shift amount(Hall data value (hx, hy)) by the optical image stabilizer.

<Image Stabilization Processing of Imaging Device>

An example of the image stabilization processing executed in an imagingmethod by the imaging device 11 will be described with reference to theflowchart in FIG. 15.

For example, in the imaging device 11, when the imaging unit 21 startsimaging of one frame, the image stabilization processing is started. Instep S11, the OIS driver 15 acquires the angular velocity data outputfrom the motion sensor 14.

In step S12, the OIS driver 15 calculates the moving amount for movingthe lens unit 12 on the basis of the angular velocity data acquired instep S11, and supplies the moving amount to the OIS actuator 16.

In step S13, the OIS actuator 16 drives the lens unit 12 according tothe moving amount supplied from the OIS driver 15 in step S12, therebyperforming the optical image stabilization.

In step S14, the OIS actuator 16 detects the position of the lens unit12 driven in step S13, and supplies the positional information of thelens unit 12 to the OIS driver 15. Then, the OIS driver 15 supplies thepositional information of the lens unit 12 and the angular velocity dataacquired in step S11 to the logic unit 22 of the image sensor 13.

In step S15, the logic unit 22 adds the positional information of thelens unit 12 supplied from the OIS driver 15 in step S14 and the angularvelocity data together with the H line counter of the image datacorresponding to reception timing of the positional information and theangular velocity data to the image data output from the imaging unit 21,and supplies the image data to the signal processing circuit 17.

In step S16, the signal processing circuit 17 performs the electronicimage stabilization processing for the image data supplied in step S15according to the function for converting the position for eachcoordinate of the image data synchronized with the positionalinformation of the lens unit 12 and the angular velocity data, using thepositional information and the angular velocity data. Thereafter, theprocessing is terminated, and similar processing is repeatedly performedevery time the next imaging of one frame is started by the imaging unit21. Note that the correction processing is continuously performedwithout termination, in capturing of a moving image and the like, apreview screen, or continuous capturing of still images for which imagestabilization is continuously performed. Furthermore, the processingfrom steps S11 to S14 is continuously performed at a preset samplingfrequency.

As described above, the imaging device 11 can suppress occurrence of ablur during the exposure time by the optical image stabilization underthe control of the OIS driver 15, and can suppress the influence ofcamera shake on an image and reliably correct the blur by the electronicimage stabilization processing by the signal processing circuit 17.

Correction results of an image captured by the imaging device 11 by suchan imaging method will be described with reference to FIG. 16.

For example, it is assumed that an angle that can be corrected by theoptical image stabilizer is ±1.5 degrees and an angle that can becorrected by the electronic image stabilization is ±6 degrees. At thistime, the correction result by the optical image stabilizer (OIS) withrespect to the vibration as illustrated in FIG. 16 shows suppression ofthe blur during the exposure time by correcting only a high frequencycomponent. Then, the correction result of the optical image stabilizer(OIS) and the electronic image stabilization (EIS) can be maintained atalmost 0 degrees by correcting a low frequency component.

As described above, the imaging device 11 can perform imaging whileperforming the correction processing by the optical image stabilizer,and can perform the electronic image stabilization for the capturedimage, using the positional information of the lens unit 12 (informationof the optical image stabilizer) and the angular velocity data. Thereby,the imaging device 11 can perform the image stabilization correspondingto the difference in the moving amount depending on the position in animage plane or the difference in the moving amount due to deviation ofthe exposure timing in one screen.

Therefore, the imaging device 11 can correct the influence of theperipheral deformation, lens distortion, and the rolling shutter and canaccurately correct the camera shake from the center to the peripherywhile suppressing occurrence of the blur during the exposure time notonly in imaging performed outdoors during the daytime or the like with ashort exposure time but also in imaging performed in a dark place,during the night time, or the like with a long exposure time.

Furthermore, in general, to increase a correction range in the opticalimage stabilizer, the device needs to be made large or large power isrequired for control. Thus, it has been difficult to increase thecorrection range. In contrast, the imaging device 11 can performcorrection of a wider range by compensating for a range not corrected bythe optical image stabilizer by the electronic image stabilization.Moreover, it is difficult for the optical image stabilizer to performcorrection in a rotation direction, whereas the imaging device 11 canperform correction in the rotation direction.

Note that FIG. 16 illustrates an example of a technique for stopping allthe vibrations by the electronic image stabilization for easyunderstanding of the effects of the image stabilization. In addition,examples of the technique for suppressing the vibration by theelectronic image stabilization include a technique for attenuating thevibration to the extent that movement of a moving image becomes smooth,a technique for not stopping the vibration at all, and other techniques,and each technique can be appropriately selected.

<Configuration Example of Imaging Device to Which Present Technology isApplied>

FIG. 17 is a block diagram illustrating a configuration example of anembodiment of an imaging device to which the present technology isapplied. Note that, in an imaging device 11A illustrated in FIG. 17,configurations common to the imaging device 11 in FIG. 13 are denoted bythe same reference numerals and detailed description of theconfigurations is omitted.

As shown in FIG. 17, the imaging device 11A includes the lens unit 12,the motion sensor 14, the OIS actuator 16, the signal processing circuit17, the display 18, the recording medium 19, and the imaging unit 21,similarly to the imaging device 11 in FIG. 13.

Then, in the imaging device 11A, a logic unit 22A of an image sensor 13Aand an OIS driver 15A have different configurations from those of theimaging device 11 in FIG. 13. The logic unit 22A has a function togenerate an OIS control signal to be described below in addition to thefunction of the logic unit 22 in FIG. 13, and further includes anexposure control unit 31 and a vibration state determination unit 32.

The logic unit 22A generates the OIS control signal for giving aninstruction on execution or stop of the optical image stabilizeraccording to exposure timing when the imaging unit 21 performs exposure,and supplies the OIS control signal to the OIS driver 15A. In a casewhere the logic unit 22A determines that a non-exposure periodcalculated from the exposure timing is longer than a predeterminedthreshold period (a time required for resetting a relative positionalrelationship between the position of the lens unit 12 and the positionof the imaging unit 21 (returning the relative positional relationshipto the center) and next starting the OIS control), the logic unit 22Agenerates the OIS control signal for resetting the OIS control duringthe non-exposure period and starting the OIS control immediately beforethe start of exposure. As described above, the processing of generatingthe OIS control signal according to the exposure timing of the imagingunit 21 is favorably performed in the logic unit 22A built in the imagesensor 13A.

For example, the logic unit 22A generates the OIS control signal on thebasis of the exposure end (read end) timing of the imaging unit 21, theexposure start timing of the next frame, and the threshold period.Furthermore, the logic unit 22A can specify the exposure start timing ofthe next frame on the basis of information such as a time between framesand an exposure time (that changes according to capture conditions suchas an automatic exposure function) of the next frame Since theaforementioned timing is determined and operated inside the image sensor13A, the OIS control signal can be more easily generated in the logicunit 22A than a configuration in which the OIS control signal isgenerated outside the image sensor 13A.

In a case where the OIS control signal gives an instruction on stop ofthe optical image stabilizer on the basis of the OIS control signalsupplied from the logic unit 22A, the OIS driver 15A performs anoperation of returning the lens unit 12 to the center position.

The imaging device 11A thus configured can perform the center returnprocessing by the optical image stabilizer between frames when the logicunit 22A supplies the OIS control signal to the OIS driver 15A. Thereby,the imaging device 11A can perform the optical image stabilizer whileresetting the lens position between frames, thereby performing, for eachframe, correction using the entire range of angles correctable by theoptical image stabilizer on a steady basis.

That is, in a case where vibration of amplitude exceeding an anglecorrectable by the optical image stabilizer occurs, the imaging device11 in FIG. 13 cannot suppress the blur during the exposure time, duringthe vibration in the exceeding range. In contrast, the imaging device11A performs the center return processing by the optical imagestabilizer, thereby suppressing occurrence of the blur during theexposure time if the vibration in one frame falls within the anglecorrectable by the optical image stabilizer.

For example, in a case where the exposure time of the imaging unit 21 isset long and the non-exposure period is shorter than the thresholdperiod, the logic unit 22A does not give an instruction on resetting ofthe relative positional relationship between the position of the lensunit 12 and the position of the imaging unit 21. Meanwhile, in a casewhere the exposure time of the imaging unit 21 is set such that thenon-exposure period becomes longer than the threshold period, the logicunit 22A gives an instruction on resetting of the relative positionalrelationship between the position of the lens unit 12 and the positionof the imaging unit 21. By controlling the exposure time of the imagingunit 21 as described above, the logic unit 22A can control the OIScontrol signal.

Therefore, in the case where the exposure time is long and the centerreturn processing by the optical image stabilizer is not performed inthe logic unit 22A, the imaging device 11A determines a vibration statein a case where the vibration in one frame exceeds the angle correctableby the optical image stabilizer and controls the exposure time, therebyperforming the center return processing by the optical image stabilizer.Thereby, the image stabilization by the optical image stabilizer worksin each frame, and the imaging device 11A can prevent occurrence of theexposure blur.

That is, the vibration state determination unit 32 of the logic unit 22Adetermines the vibration state of the imaging device 11A on the basis ofthe lens positional information (Hall data) of the optical imagestabilizer, a gyro signal (angular velocity data) output from the motionsensor 14, and the like. For example, the vibration state determinationunit 32 determines that the vibration state is a first vibration statein a case where the vibration in one frame is in a gentle state withinthe correctable range by the optical image stabilizer, and determinesthat the vibration state is a second vibration state (a first exposuretime control mode to be described below) in a case where the vibrationis in a state exceeding the correctable range. Moreover, the vibrationstate determination unit 32 determines that the vibration state is athird vibration state (a second exposure time control mode to bedescribed below) in a case where the vibration in one frame is in a moreintense state than the vibration in the second vibration state.

Then, the exposure control unit 31 of the logic unit 22A performsexposure control for the imaging unit 21 under the imaging conditionsaccording to the determination result of the vibration state of theimaging device 11A by the vibration state determination unit 32. Forexample, the exposure control unit 31 determines the exposure time inwhich the imaging unit 21 performs exposure and sets the determinedexposure time for the imaging unit 21, and determines a gain (analoggain and digital gain) for the image acquired by the imaging unit 21 inthe exposure time and sets the gain for the imaging unit 21.

Note that the function to adopt a maximum value of the exposure timeallowed within a frame rate as a maximum exposure time regardless of thevibration state, and obtain an optimum exposure time and an optimum gainsuch that an image with optimum brightness can be captured by theimaging unit 21, on the basis of the image output from the imaging unit21, information of an illuminance sensor (not illustrated), and thelike, is included in the imaging device 11 in FIG. 13 although noillustration and description are given. Since the present technology ischaracterized in controlling the exposure time according to thevibration state, description is given as an individual function in theimaging device 11A.

In the first vibration state, the exposure control unit 31 obtains theoptimum exposure time and the optimum gain in which an image withoptimum brightness can be captured, and sets the obtained optimumexposure time and optimum gain for the imaging unit 21 as the imagingconditions. For example, the exposure control unit 31 adopts the maximumvalue of the exposure time as the maximum exposure time allowed within aframe rate, and can obtain the optimum exposure time and the optimumgain such that an image with optimum brightness can be captured by theimaging unit 21, on the basis of the image output from the imaging unit21, the information of an illuminance sensor (not illustrated), and thelike.

In the second vibration state, the exposure control unit 31 adopts afirst threshold exposure time (see FIG. 20) as a maximum value, in whichthe non-exposure period can be secured, in which the relative positionalrelationship between the position of the lens unit 12 and the positionof the imaging unit 21 can be reset (returned to the center), determinesan exposure time and a gain, and sets the determined exposure time andgain for the imaging unit 21. For example, in a case where the optimumexposure time in the first vibration state is longer than the firstthreshold exposure time, the exposure control unit 31 sets the firstthreshold exposure time as the exposure time. Then, the exposure controlunit 31 increases the gain from the optimum gain to compensate for theshortage of brightness due to the shortened exposure time, and obtains again such that an image with brightness can be obtained, the brightnessbeing equivalent to the case where an image is captured with the optimumexposure time and optimum gain in the first vibration state. Meanwhile,in a case where the optimum exposure time in the first vibration stateis shorter than the first threshold exposure time, the exposure controlunit 31 uses the optimum exposure time and the optimum gain in the firstvibration state as they are.

Moreover, in the third vibration state, the exposure control unit 31adopts a second threshold exposure time (see FIG. 22) shorter than thefirst threshold exposure time as the maximum value, determines anexposure time and a gain, and sets the determined exposure time and gainfor the imaging unit 21. For example, in a case where the optimumexposure time in the first vibration state is longer than the secondthreshold exposure time, the exposure control unit 31 sets the secondthreshold exposure time as the exposure time. Then, the exposure controlunit 31 increases the gain from the optimum gain to compensate for theshortage of brightness due to the shortened exposure time, and obtains again such that an image with brightness can be obtained, the brightnessbeing equivalent to the case where an image is captured with the optimumexposure time and optimum gain in the first vibration state. Meanwhile,in a case where the optimum exposure time in the first vibration stateis shorter than the second threshold exposure time, the exposure controlunit 31 uses the optimum exposure time and the optimum gain in the firstvibration state as they are.

Here, determination of the vibration state by the vibration statedetermination unit 32, exposure control by the exposure control unit 31,and OIS control information generated by the logic unit 22A will bedescribed with reference to FIGS. 18 to 22.

FIG. 18 illustrates an example of the exposure control and the OIScontrol information in the first vibration state in which vibration inone frame falls within a range (no blur) correctable by the opticalimage stabilizer.

For example, in a case where the proper exposure time calculated on thebasis of the image output from the imaging unit 21, the information ofan illuminance sensor (not illustrated), and the like is longer than thefirst threshold exposure time, imaging is performed using control of thenormal optical image stabilizer with the proper exposure time when thevibration is gentle. Therefore, as illustrated in FIG. 18, in the firstvibration state where the vibration is gentle within the correctablerange by the optical image stabilizer, the vibration and a correctedamount by the optical image stabilizer overlap, and the exposure blurdoes not occur. Furthermore, at this time, exposure is performed withthe optimum exposure time, and the logic unit 22A outputs the OIScontrol information (OIS enable) for giving an instruction on executionof the optical image stabilizer on a steady basis and does not performthe center return processing by the optical image stabilizer.

In contrast, as illustrated in FIG. 19, in a case where the vibration isso intense as to exceed the correctable range by the optical imagestabilizer, the vibration cannot be completely corrected by the opticalimage stabilizer and the exposure blur occurs.

Therefore, when the vibration state as in FIG. 19 is determined to bethe second vibration state, the exposure control unit 31 sets the firstthreshold exposure time as the exposure time and performs imaging.Thereby, as illustrated in FIG. 20, the logic unit 22A outputs the OIScontrol information (OIS enable) for giving an instruction on executionof the optical image stabilizer in the period where exposure isperformed, and outputs the OIS control information (OIS disable) forgiving an instruction on stop of the optical image stabilizer in theperiod where no exposure is performed.

Thereby, the control of resetting the relative positional relationshipbetween the position of the lens unit 12 and the position of the imagingunit 21 in the non-exposure period, and executing the optical imagestabilizer in the exposure period can be performed. Therefore, even inthe vibration exceeding the correctable range by the optical imagestabilizer, as illustrated by the broken line in FIG. 20, occurrence ofthe exposure blur can be avoided and an image without a blur by theoptical image stabilizer can be captured.

Note that, since a change in the position and deformation of the imageare caused between frames as the output result of the optical imagestabilizer at this rate, the image stabilization technology describedwith reference to the imaging device 11 in FIG. 13 is used. That is, theinfluence of the motion of the imaging unit 21 on the image is correctedaccording to a function for converting the position on the basis ofrelative positional information with respect to the lens unit 12 or theimaging unit 21 synchronized for each coordinate on the image and motioninformation indicating the motion of the physically detected imagingunit 21. Thereby, the imaging device 11A can correct the positionaldeviation or deformation between frames and can obtain an image in whichthe influence of the positional deviation, deformation, and exposureblur due to the camera shake is corrected.

At this time, it is possible to prevent the image from becoming dark bycapturing the image with an increased gain (sensitivity) by the amountof a shortened exposure time. However, as a result, noise slightlyincreases with the increase in the gain. However, it is possible tofurther improve overall image quality by suppressing deterioration ofthe image quality due to the exposure blur caused by intense vibrationrather than suppressing deterioration of the image quality due to theslight increase in noise with the increase in the gain.

Note that, in the case where the proper exposure time calculated on thebasis of the image output from the imaging unit 21, the information ofan illuminance sensor (not illustrated), and the like is shorter thanthe first threshold exposure time, the logic unit 22A outputs the OIScontrol information (OIS enable) for giving an instruction on executionof the optical image stabilizer in the period where exposure isperformed, and outputs the OIS control information (OIS disable) forgiving an instruction on stop of the optical image stabilizer in theperiod where no exposure is performed, regardless of whether thevibration is gentle or intense. As a result, the center returnprocessing by the optical image stabilizer is performed, and correctioncan be performed such that no exposure blur occurs.

Therefore, the imaging device 11A can capture an image without a blur bythe optical image stabilizer even when the optimum exposure time islonger than the first threshold exposure time, and can correct theinfluence of the motion of the imaging unit 21 on the image, similarlyto the imaging device 11 in FIG. 13. That is, the imaging device 11A canacquire an image without positional deviation, peripheral deformation,distortion due to the rolling shutter, deformation due to the influenceof lens distortion, and the like.

Moreover, as illustrated in FIG. 21, in the case of the vibration sointense as to exceed the correctable range by the optical imagestabilizer during the exposure period of one frame according to thefirst threshold exposure time, correction by the optical imagestabilizer exceeds a correction limit during the exposure period even ifthe center return processing by the optical image stabilizer isperformed, and thus the exposure blur occurs.

Therefore, when the vibration state as in FIG. 21 is determined to bethe third vibration state, the exposure control unit 31 sets the secondthreshold exposure time as the exposure time and performs imaging withthe second threshold exposure time further shorter than the firstthreshold exposure time (FIG. 21), as illustrated in FIG. 22. Byperforming the imaging with the second threshold exposure time asdescribed above, the vibration amount during the exposure period becomessmall. Therefore, the correction by the optical image stabilizer doesnot exceed the correction limit during the exposure period, and an imagewithout a blur by the optical image stabilizer can be captured whileavoiding occurrence of the exposure blur.

Note that, in the imaging device 11A, in a case where the analog gain orthe digital gain in the image sensor 13A is not sufficient, the signalprocessing circuit 17 may adjust the insufficient gain. Furthermore, inthe present embodiment, the optimum exposure time and the optimum gain,and the exposure time and the gain adjusted according to thedetermination result by the vibration state determination unit 32 aredistinguished.

Then, the imaging device 11A performs imaging while performing thecenter return processing by the optical image stabilizer as describedabove, and applies the electronic image stabilization based on theangular velocity information and the relative positional informationbetween the position of the lens unit 12 and the position of the imagingunit 21 to the image without the exposure blur corrected by the opticalimage stabilizer, similarly to the imaging device 11 in FIG. 13, therebycorrecting the influence of the motion of the imaging unit 21 on theimage. Thereby, the imaging device 11A can acquire an image without thepositional deviation, peripheral deformation, distortion due to therolling shutter, deformation due to the influence of lens distortion,and the like without occurrence of the exposure blur.

<First Exposure Time Control Mode>

Switching of the imaging conditions in the first exposure time controlmode will be described with reference to FIG. 23. Here, in the firstexposure time control mode, the vibration state determination unit 32determines whether the vibration of the imaging device 11A is the firstvibration state or the second vibration state.

The vibration state determination unit 32 determines, for each oneframe, whether or not the vibration state of the imaging device 11A hasreached a predetermined condition, and notifies the exposure controlunit 31 of a determination result based on the determination.

For example, in the first exposure time control mode in the imagingdevice 11A, reaching a situation where a state where the vibration ofthe imaging device 11A exceeds a first threshold has occurred apredetermined number of times or more during a predetermined period, isset as a first condition for determining the vibration state of theimaging device 11A. Furthermore, reaching a situation where a statewhere the vibration of the imaging device 11A falls below a secondthreshold has occurred a predetermined number of times or more insuccession, is set as a second condition for determining the vibrationstate of the imaging device 11A.

Furthermore, as the imaging conditions in the first vibration state, theoptimum exposure time and the optimum gain are used. Here, the maximumvalue of the exposure time is adopted as the maximum exposure timeallowed within a frame rate, and the optimum exposure time is obtainedsuch that an image with optimum brightness can be captured by theimaging unit 21 on the basis of the image output from the imaging unit21, the information of an illuminance sensor (not illustrated), and thelike. Furthermore, the optimum gain is obtained to optimize thebrightness of the image captured with the optimum exposure time.

Moreover, as imaging conditions in a second vibration state, in a casewhere the optimum exposure time in the first vibration state is longerthan a first threshold exposure time, the exposure time is set as thefirst threshold exposure time, and the value of the gain is setaccording to the exposure time. Alternatively, in a case where theoptimum exposure time in the first vibration state is shorter than thefirst threshold exposure time, the optimum exposure time and the optimumgain in the first vibration state are set as they are as the imagingconditions (an imaging condition determination rule in the secondvibration state). The first threshold exposure time is determined tosecure the non-exposure period in which the relative positionalrelationship between the position of the lens unit 12 and the positionof the imaging unit 21 can be reset (can be returned to the center).Furthermore, in the case where the exposure time becomes shorter thanthe optimum exposure time, the gain is increased from the optimum gainto compensate for the shortage of brightness due to the shortenedexposure time, and is obtained to be able to acquire an image withbrightness equivalent to an image captured with the optimum exposuretime and optimum gain.

First, in the imaging device 11A, use of the first vibration state as aninitial setting is set in the vibration state determination unit 32 atthe start of imaging, and the vibration state determination unit 32notifies the exposure control unit 31 of the first vibration state as adetermination result of the vibration state of the imaging device 11A.Accordingly, the exposure control unit 31 sets the above-describedoptimum exposure time and optimum gain for the imaging unit 21 as theimaging conditions in the first vibration state, and the imaging unit 21performs imaging

Then, in a case where the vibration state determination unit 32continues determination that the vibration state of the imaging device11A has not reached the first condition in the first vibration state,the imaging unit 21 in the imaging device 11A performs imaging under theunchanged imaging conditions in the first vibration state.

Meanwhile, in a case where the vibration state determination unit 32determines that the vibration state of the imaging device 11A hasreached the first condition in the first vibration state, the vibrationstate determination unit 32 notifies the exposure control unit 31 of thedetermination result indicating that the vibration state hastransitioned from the first vibration state to the second vibrationstate. Accordingly, the exposure control unit 31 sets, for the imagingunit 21, the exposure time and the gain based on the imaging conditiondetermination rule in the second vibration state, and the imaging unit21 performs imaging.

Then, in a case where the vibration state determination unit 32continues determination that the vibration state of the imaging device11A has not reached the second condition in the second vibration state,the imaging unit 21 in the imaging device 11A performs imaging under theunchanged imaging conditions based on the imaging conditiondetermination rule in the second vibration state.

Meanwhile, in a case where the vibration state determination unit 32determines that the vibration state of the imaging device 11A hasreached the second condition in the second vibration state, thevibration state determination unit 32 notifies the exposure control unit31 of the determination result indicating that the vibration state hastransitioned from the second vibration state to the first vibrationstate. Accordingly, the exposure control unit 31 sets theabove-described optimum exposure time and optimum gain for the imagingunit 21 as the imaging conditions in the first vibration state, and theimaging unit 21 performs imaging

By switching the imaging conditions in the first exposure time controlmode to control the exposure time, the OIS control information changesas illustrated in FIG. 18 or 20. That is, when the vibration statebecomes the second vibration state and the exposure time becomes equalto or less than the first threshold exposure time, the logic unit 22Aoutputs the OIS control information (OIS disable) for giving aninstruction on stop of the optical image stabilizer in the non-exposureperiod. Accordingly, the OIS driver 15A performs processing of resettingthe relative positional relationship between the position of the lensunit 12 and the position of the imaging unit 21 and returning thepositions to the center positions. Next, the logic unit 22A switches theOIS control information to the OIS control information (OIS enable) forgiving an instruction on execution of the optical image stabilizerimmediately before the start of the exposure time of the next frame.Accordingly, the OIS driver 15A restarts the optical image stabilizer.

By performing the exposure control according to the vibration state asdescribed above, an image in which no exposure blur occurs in thecapture result can be obtained, as described with reference to FIG. 20,even in the case where vibration exceeding the correctable range by theoptical image stabilizer occurs, as illustrated in FIG. 19.

That is, in the case where the exposure control according to thevibration state is not performed and the vibration exceeding thecorrectable range by the optical image stabilizer occurs, the opticalimage stabilizer cannot be performed, and the exposure blur due to thevibration during exposure occurs in the capture result. In contrast, inthe case where the exposure control according to the vibration state isperformed and the vibration exceeding the correctable range by theoptical image stabilizer occurs, the relative positional relationshipbetween the position of the lens unit 12 and the position of the imagingunit 21 is reset in the non-exposure period by setting the exposure timeto equal to or less than the first threshold exposure time. Thereby, theimage stabilization can be performed over the entire correctable rangeby the optical image stabilizer at the time of exposure, and an image inwhich occurrence of the exposure blur is avoided can be obtained in thecapture result unless the vibration in one frame exceeds the correctablerange by the optical image stabilizer.

Note that, some increase in noise is assumed with the imaging with anincreased gain in accordance with the shortened exposure time in theimaging device 11A. However, the influence is smaller than thedeterioration of the image quality due to occurrence of the exposureblur, and an image with higher image quality can be obtained.

Moreover, in a case where the vibration state determination unit 32determines that the vibration state of the imaging device 11A hasreached the second condition in the second vibration state, thevibration state determination unit 32 notifies the exposure control unit31 of the determination result indicating that the vibration state hastransitioned from the second vibration state to the first vibrationstate. Accordingly, the exposure control unit 31 controls exposure ofthe imaging unit 21 to perform imaging under the imaging conditions inthe first vibration state.

That is, when the vibration of the imaging device 11A becomes gentle,the imaging device 11A can correct the vibration in the correctablerange by the optical image stabilizer even if the exposure time is madelonger than the first threshold exposure time, as illustrated in FIG.18. Thereby, an image in which no exposure blur occurs and no increasein noise can be obtained in the capture result.

The exposure time control in the first exposure time control mode asdescribed with reference to FIG. 23 will be described with reference tothe flowchart in FIG. 24.

For example, in the imaging device 11A, the image stabilizationprocessing is started when the imaging unit 21 starts imaging. In stepS21, the vibration state determination unit 32 notifies the exposurecontrol unit 31 that the vibration state is the first vibration state,and the exposure control unit 31 sets the imaging conditions in thefirst vibration state for the imaging unit 21 as initial settings.

In step S22, the vibration state determination unit 32 determineswhether or not the vibration state of the imaging device 11A has reachedthe first condition. In step S22, the processing stands by until thevibration state determination unit 32 determines that the vibrationstate of the imaging device 11A has reached the first condition, and theimaging unit 21 performs imaging under the unchanged imaging conditionsin the first vibration state.

On the other hand, in step S22, in the case where the vibration statedetermination unit 32 determines that the vibration state of the imagingdevice 11A has reached the first condition, the processing proceeds tostep S23.

In step S23, the vibration state determination unit 32 notifies theexposure control unit 31 of the determination result indicating that thevibration state has transitioned from the first vibration state to thesecond vibration state. Accordingly, the exposure control unit 31determines the imaging conditions on the basis of the imaging conditiondetermination rule in the second vibration state and performs exposurecontrol for the imaging unit 21, and the imaging unit 21 performsimaging.

In step S24, the vibration state determination unit 32 determineswhether or not the vibration state of the imaging device 11A has reachedthe second condition. In step S24, the processing stands by until thevibration state determination unit 32 determines that the vibrationstate of the imaging device 11A has reached the second condition, andthe imaging unit 21 performs imaging under the unchanged imagingconditions based on the imaging condition determination rule in thesecond vibration state.

On the other hand, in step S24, in the case where the vibration statedetermination unit 32 determines that the vibration state of the imagingdevice 11A has reached the second condition, the processing proceeds tostep S25.

In step S25, the vibration state determination unit 32 notifies theexposure control unit 31 of the determination result indicating that thevibration state has transitioned from the second vibration state to thefirst vibration state. Accordingly, the exposure control unit 31performs exposure control for the imaging unit 21 to change the imagingconditions to the imaging conditions in the first vibration state, andthe imaging unit 21 performs imaging under the imaging conditions of thefirst vibration state.

After the processing in step S25, the processing returns to step S22,and hereinafter, similar processing is repeatedly performed until theimaging is terminated.

Here, a method of determining the vibration state by the vibration statedetermination unit 32 will be described.

For example, the vibration state determination unit 32 can determinewhether or not the vibration state is within a set threshold, using lenspositional information (for example, Hall data) of the optical imagestabilizer.

In the first vibration state, the vibration state determination unit 32determines that the vibration state transitions to the second vibrationstate when the value of the Hall data has exceeded a first rate Tp1%(Tp1≤100) with respect to a movable range as the optical imagestabilizer by a first determination frame number Tc1 (Tf1≥Tc1≥1) in afirst past frame number Tf1 (Tf1≥1). Meanwhile, in the second vibrationstate, the vibration state determination unit 32 determines that thevibration state transitions to the first vibration state when a peakvalue of the Hall data has fallen below a second rate Tp2% (Tp2<100)with respect to the movable range as the optical image stabilizer by asecond determination frame number Tc2 (Tc2≥1) in succession. Forexample, the first rate Tp1=75, the first past frame number Tf1=10, thefirst determination frame number Tc1=3, the second rate Tp2=25, and thesecond determination frame number Tc2=5 can be set.

Furthermore, the vibration state determination unit 32 can determinewhether or not the vibration state is within the set threshold, using anintegral angle of the vibration calculated from an angular velocity(gyro signal) and the like output by the motion sensor 14.

In the first vibration state, the vibration state determination unit 32determines that the vibration state transitions to the second vibrationstate when the integral angle (angle to be corrected) in one frame hasexceeded the first rate Tp1% (Tp1≤100) of the angle movable as theoptical image stabilizer by the first determination frame number Tc1(Tf1≥Tc1≥1) in the first past frame number (Tf1≥1). Meanwhile, in thesecond vibration state, the vibration state determination unit 32determines that the vibration state transitions to the first vibrationstate when the change amount (angle to be corrected) of the integralangle in one frame has fallen below the second rate Tp2% (Tp2<100) ofthe angle movable as the optical image stabilizer by the seconddetermination frame number Tc2 (Tc2≥1) in succession. For example, thefirst rate Tp1=75, the first past frame number Tf1=10, the firstdetermination frame number Tc1=3, the second rate Tp2=25, and the seconddetermination frame number Tc2=5 can be set.

Moreover, the vibration state determination unit 32 can determinewhether or not the vibration state is within the set threshold, usingthe angular velocity (gyro signal) output by the motion sensor 14.

In the first vibration state, the vibration state determination unit 32determines that the vibration state has changed to the second vibrationstate when a peak value of the angular velocity sampled in one frame hasexceeded a preset first angular velocity Thω1 (degree per second) by thefirst determination frame number Tc1 (Tf1≥Tc1≥1) in the first past framenumber Tf1 (Tf1≥1). Meanwhile, in the second vibration state, thevibration state determination unit 32 determines that the vibrationstate transitions to the first vibration state when the peak value ofthe angular velocity sampled in one frame has fallen below a presetsecond angular velocity Thω2 (degree per second) by the seconddetermination frame number Tc2 (Tc2≥1) in succession. For example, thefirst angular velocity Thω1=20, the first past frame number Tf1=10, thefirst determination frame number Tc1=3, the second angular velocityThω2=10, and the second determination frame number Tc2=10 can be set.

In addition, the vibration state determination unit 32 may determinewhether or not the vibration state is within the set threshold, using acombination of the above-described Hall data, the integral angle (or acorrected angle), and the angular velocity. Furthermore, an averagevalue may be used instead of the peak value within a frame, anddetermination may be performed using a predicted value from data of pastseveral frames. Moreover, in determining whether or not the vibrationstate is within the set threshold, the vibration state determinationunit 32 can make determination for each axis (pitch or yaw) or using themagnitude of a vector sum of the axes.

<Second Exposure Time Control Mode>

Change of the imaging conditions in the second exposure time controlmode will be described with reference to FIG. 25. Here, in the secondexposure time control mode, the vibration state determination unit 32determines whether the vibration of the imaging device 11A is the firstvibration state, the second vibration state, or the third vibrationstate. Note that, here, as the second exposure time control mode, thethree vibration states (three values) of the first vibration state, thesecond vibration state, and the third vibration state will be described.However, similarly, the vibration state may be classified into four ormore vibration states (multivalued).

In the second exposure time control mode, the vibration statedetermination unit 32 determines, for each one frame, whether or not thevibration state of the imaging device 11A has reached a predeterminedcondition, and notifies the exposure control unit 31 of a determinationresult based on the determination, similarly to the above-describedfirst exposure time control mode.

For example, in the second exposure time control mode in the imagingdevice 11A, reaching a situation where a state where the vibration ofthe imaging device 11A exceeds a first threshold in the first vibrationstate has occurred a predetermined number of times or more during apredetermined period, is set as a first condition for determining thevibration state of the imaging device 11A. Furthermore, reaching asituation where a state where the vibration exceeds a second thresholdhas occurred a predetermined number of times or more during apredetermined period although the first condition is not satisfied inthe first vibration state, is set as a second condition for determiningthe vibration state of the imaging device 11A (the second threshold<thefirst threshold). Furthermore, reaching a situation where a state wherethe vibration of the imaging device 11A exceeds a third threshold hasoccurred a predetermined number of times or more during a predeterminedperiod in the second vibration state, is set as a third condition fordetermining the vibration state of the imaging device 11A.

Moreover, in the second exposure time control mode in the imaging device11A, reaching a situation where a state where the vibration of theimaging device 11A falls below a fourth threshold in the secondvibration state has occurred a predetermined number of times or more insuccession, is set as a fourth condition for determining the vibrationstate of the imaging device 11A. Furthermore, reaching a situation wherea state where the vibration of the imaging device 11A falls below afifth threshold in the third vibration state has occurred apredetermined number of times or more in succession, is set as a fifthcondition for determining the vibration state of the imaging device 11A.Furthermore, reaching a situation where a state where the vibration ofthe imaging device 11A falls below a sixth threshold has occurred apredetermined number of times or more in succession although the fifthcondition is not satisfied in the third vibration state, is set as asixth condition for determining the vibration state of the imagingdevice 11A (the fifth threshold<the sixth threshold).

Furthermore, as the imaging conditions in the first vibration state, theoptimum exposure time and the optimum gain are used. Here, the maximumvalue of the exposure time is adopted as the maximum exposure timeallowed within a frame rate, and the optimum exposure time is obtainedsuch that an image with optimum brightness can be captured by theimaging unit 21 on the basis of the image output from the imaging unit21, the information of an illuminance sensor (not illustrated), and thelike. Furthermore, the optimum gain is obtained to optimize thebrightness of the image captured with the optimum exposure time.

Moreover, as imaging conditions in a second vibration state, in a casewhere the optimum exposure time in the first vibration state is longerthan a first threshold exposure time, the exposure time is set as thefirst threshold exposure time, and the value of the gain is setaccording to the exposure time. Alternatively, in a case where theoptimum exposure time in the first vibration state is shorter than thefirst threshold exposure time, the optimum exposure time and the optimumgain in the first vibration state are set as they are as the imagingconditions (an imaging condition determination rule in the secondvibration state). Here, the first threshold exposure time is determinedto secure the non-exposure period in which the relative positionalrelationship between the position of the lens unit 12 and the positionof the imaging unit 21 can be reset (can be returned to the center).Furthermore, in the case where the exposure time becomes shorter thanthe optimum exposure time, the gain is increased from the optimum gainto compensate for the shortage of brightness due to the shortenedexposure time, and is obtained to be able to acquire an image withbrightness equivalent to an image captured with the optimum exposuretime and optimum gain.

Furthermore, as imaging conditions in the third vibration state, in acase where the optimum exposure time in the first vibration state islonger than the second threshold exposure time, the exposure time is setas the second threshold exposure time, and the value of the gain is setaccording to the exposure time. Alternatively, in a case where theoptimum exposure time in the first vibration state is shorter than thesecond threshold exposure time, the optimum exposure time and theoptimum gain in the first vibration state are set as they are as theimaging conditions (an imaging condition determination rule in the thirdvibration state). Here, the second threshold exposure time is determinedto be shorter than the first threshold exposure time. Furthermore, inthe case where the exposure time becomes shorter than the optimumexposure time, the gain is increased from the optimum gain to compensatefor the shortage of brightness due to the shortened exposure time, andis obtained to be able to acquire an image with brightness equivalent toan image captured with the optimum exposure time and optimum gain.

First, in the imaging device 11A, use of the first vibration state as aninitial setting is set in the vibration state determination unit 32 atthe start of imaging, and the vibration state determination unit 32notifies the exposure control unit 31 of the first vibration state as adetermination result of the vibration state of the imaging device 11A.Accordingly, the exposure control unit 31 sets the above-describedoptimum exposure time and optimum gain for the imaging unit 21 as theimaging conditions in the first vibration state, and the imaging unit 21performs imaging

Then, in a case where the vibration state determination unit 32continues determination that the vibration state of the imaging device11A has not reached the first and second conditions in the firstvibration state, the imaging unit 21 in the imaging device 11A performsimaging under the unchanged imaging conditions in the first vibrationstate.

Meanwhile, in a case where the vibration state determination unit 32determines that the vibration state of the imaging device 11A hasreached the first condition in the first vibration state, the vibrationstate determination unit 32 notifies the exposure control unit 31 of thedetermination result indicating that the vibration state hastransitioned from the first vibration state to the third vibrationstate. Accordingly, the exposure control unit 31 sets, for the imagingunit 21, the exposure time and the gain based on the imaging conditiondetermination rule in the third vibration state, and the imaging unit 21performs imaging.

Meanwhile, in a case where the vibration state determination unit 32determines that the vibration state of the imaging device 11A hasreached the second condition in the first vibration state, the vibrationstate determination unit 32 notifies the exposure control unit 31 of thedetermination result indicating that the vibration state hastransitioned from the first vibration state to the second vibrationstate. Accordingly, the exposure control unit 31 sets, for the imagingunit 21, the exposure time and the gain based on the imaging conditiondetermination rule in the second vibration state, and the imaging unit21 performs imaging.

Furthermore, in a case where the vibration state determination unit 32continues determination that the vibration state of the imaging device11A has not reached the third and fourth conditions in the secondvibration state, the imaging unit 21 in the imaging device 11A performsimaging under the unchanged imaging conditions based on the imagingcondition determination rule in the second vibration state.

Meanwhile, in a case where the vibration state determination unit 32determines that the vibration state of the imaging device 11A hasreached the third condition in the second vibration state, the vibrationstate determination unit 32 notifies the exposure control unit 31 of thedetermination result indicating that the vibration state hastransitioned from the second vibration state to the third vibrationstate. Accordingly, the exposure control unit 31 sets, for the imagingunit 21, the exposure time and the gain based on the imaging conditiondetermination rule in the third vibration state, and the imaging unit 21performs imaging.

Meanwhile, in a case where the vibration state determination unit 32determines that the vibration state of the imaging device 11A hasreached the fourth condition in the second vibration state, thevibration state determination unit 32 notifies the exposure control unit31 of the determination result indicating that the vibration state hastransitioned from the second vibration state to the first vibrationstate. Accordingly, the exposure control unit 31 sets theabove-described optimum exposure time and optimum gain for the imagingunit 21 as the imaging conditions in the first vibration state, and theimaging unit 21 performs imaging

Furthermore, in a case where the vibration state determination unit 32continues determination that the vibration state of the imaging device11A has not reached the fifth and sixth conditions in the thirdvibration state, the imaging unit 21 in the imaging device 11A performsimaging under the unchanged imaging conditions based on the imagingcondition determination rule in the third vibration state.

Meanwhile, in a case where the vibration state determination unit 32determines that the vibration state of the imaging device 11A hasreached the fifth condition in the third vibration state, the vibrationstate determination unit 32 notifies the exposure control unit 31 of thedetermination result indicating that the vibration state hastransitioned from the third vibration state to the first vibrationstate. Accordingly, the exposure control unit 31 sets theabove-described optimum exposure time and optimum gain for the imagingunit 21 as the imaging conditions in the first vibration state, and theimaging unit 21 performs imaging

Meanwhile, in a case where the vibration state determination unit 32determines that the vibration state of the imaging device 11A hasreached the sixth condition in the third vibration state, the vibrationstate determination unit 32 notifies the exposure control unit 31 of thedetermination result indicating that the vibration state hastransitioned from the third vibration state to the second vibrationstate. Accordingly, the exposure control unit 31 sets, for the imagingunit 21, the exposure time and the gain based on the imaging conditiondetermination rule in the second vibration state, and the imaging unit21 performs imaging.

By switching the imaging conditions in the second exposure time controlmode to control the exposure time, the imaging device 11A can performexposure control such that no exposure blur occurs according to thevibration state, as described with reference to FIGS. 18 to 22.

Note that, in the case where vibration exceeding the correctable rangeby the optical image stabilizer occurs even under the vibrationconditions in the third vibration state, occurrence of the exposure blurcannot be avoided, and occurrence of a blur in the image is assumed. Inthis case, to cope with occurrence of a blur in the image, a shorterexposure time may be set as the second threshold exposure time to make avibration amount during the exposure period small. Moreover, theexposure control may be performed by finely dividing the vibration stateinto four or five stages rather than setting the imaging conditions inthree stages as illustrated in FIG. 25.

Exposure time control in the second exposure time control mode asdescribed with reference to FIG. 25 will be described with reference tothe flowchart in FIG. 26.

For example, in the imaging device 11A, the image stabilizationprocessing is started when the imaging unit 21 starts imaging. In stepS31, the vibration state determination unit 32 notifies the exposurecontrol unit 31 that the vibration state is the first vibration state,and the exposure control unit 31 sets the imaging conditions in thefirst vibration state for the imaging unit 21 as initial settings.

In step S32, the vibration state determination unit 32 determineswhether or not the vibration state of the imaging device 11A has reachedthe first condition. In a case where the vibration state determinationunit 32 determines in step S32 that the vibration state of the imagingdevice 11A has not reached the first condition, the processing proceedsto step S33.

In step S33, the vibration state determination unit 32 determineswhether or not the vibration state of the imaging device 11A has reachedthe second condition. In a case where the vibration state determinationunit 32 determines in step S33 that the vibration state of the imagingdevice 11A has not reached the second condition, the processing returnsto step S32, and similar processing is repeated. That is, the processingstands by until determination is made in step S32 that the vibrationstate has reached the first condition or in step S33 that the vibrationstate has reached the second condition, and the imaging unit 21 performsimaging under the unchanged imaging conditions in the first vibrationstate.

On the other hand, in step S33, in the case where the vibration statedetermination unit 32 determines that the vibration state of the imagingdevice 11A has reached the second condition, the processing proceeds tostep S34.

In step S34, the vibration state determination unit 32 notifies theexposure control unit 31 of the determination result indicating that thevibration state has transitioned from the first vibration state to thesecond vibration state. Accordingly, the exposure control unit 31performs exposure control for the imaging unit 21 on the basis of theimaging condition determination rule in the second vibration state, andthe imaging unit 21 performs imaging.

In step S35, the vibration state determination unit 32 determineswhether or not the vibration state of the imaging device 11A has reachedthe third condition. In a case where the vibration state determinationunit 32 determines in step S35 that the vibration state of the imagingdevice 11A has not reached the third condition, the processing proceedsto step S36.

In step S36, the vibration state determination unit 32 determineswhether or not the vibration state of the imaging device 11A has reachedthe fourth condition. In a case where the vibration state determinationunit 32 determines in step S36 that the vibration state of the imagingdevice 11A has not reached the fourth condition, the processing returnsto step S35, and hereinafter, similar processing is repeated. That is,the processing stands by until determination is made in step S35 thatthe vibration state has reached the third condition or in step S36 thatthe vibration state has reached the fourth condition, and the imagingunit 21 performs imaging under the unchanged imaging conditions based onthe imaging condition determination rule in the second vibration state.

On the other hand, in step S36, in the case where the vibration statedetermination unit 32 determines that the vibration state of the imagingdevice 11A has reached the fourth condition, the processing proceeds tostep S37.

In step S37, the vibration state determination unit 32 notifies theexposure control unit 31 of the determination result indicating that thevibration state has transitioned from the second vibration state to thefirst vibration state. Accordingly, the exposure control unit 31 setsthe above-described optimum exposure time and optimum gain for theimaging unit 21, and the imaging unit 21 performs imaging After theprocessing in step S37, the processing returns to step S32, andhereinafter, similar processing is repeatedly performed until theimaging is terminated.

Meanwhile, in a case where the vibration state determination unit 32determines in step S32 that the vibration state of the imaging device11A has reached the first condition, or in a case where the vibrationstate determination unit 32 determines in step S35 that the vibrationstate of the imaging device 11A has reached the third condition, theprocessing proceeds to step S38.

In step S38, the vibration state determination unit 32 notifies theexposure control unit 31 of the determination result indicating that thevibration state has transitioned from the first or second vibrationstate to the third vibration state. Accordingly, the exposure controlunit 31 performs exposure control for the imaging unit 21 to performimaging under the imaging conditions based on the imaging conditiondetermination rule in the third vibration state, and the imaging unit 21performs imaging.

In step S39, the vibration state determination unit 32 determineswhether or not the vibration state of the imaging device 11A has reachedthe fifth condition. In a case where the vibration state determinationunit 32 determines in step S39 that the vibration state of the imagingdevice 11A has not reached the fifth condition, the processing proceedsto step S40.

In step S40, the vibration state determination unit 32 determineswhether or not the vibration state of the imaging device 11A has reachedthe sixth condition. In a case where the vibration state determinationunit 32 determines in step S40 that the vibration state of the imagingdevice 11A has not reached the sixth condition, the processing returnsto step S39, and hereinafter, similar processing is repeated. That is,the processing stands by until determination is made in step S39 thatthe vibration state has reached the fifth condition or in step S40 thatthe vibration state has reached the sixth condition, and the imagingunit 21 performs imaging under the unchanged imaging conditions based onthe imaging condition determination rule in the third vibration state.

Meanwhile, the processing proceeds to step S34 in a case where thevibration state determination unit 32 determines in step S40 that thevibration state of the imaging device 11A has reached the sixthcondition, and hereinafter, the imaging unit 21 performs imaging underthe imaging conditions based on the imaging condition determination rulein the second vibration state, as described above.

Meanwhile, the processing proceeds to step S37 in a case where thevibration state determination unit 32 determines in step S39 that thevibration state of the imaging device 11A has reached the fifthcondition, and hereinafter, the imaging unit 21 performs imaging underthe imaging conditions in the first vibration state, as described above.

Thereafter, similar processing is repeatedly performed until the imagingis terminated.

A modification of transition of the vibration state in the secondexposure time control mode will be described with reference to FIG. 27.

In the above transition of the vibration state described with referenceto FIG. 25, it is possible to make determination to jump over the secondvibration state that is between the first vibration state and the thirdvibration state, such as transition from the first vibration state tothe third vibration state or transition from third vibration state tofirst vibration state. In contrast, as illustrated in FIG. 27, it isalso possible to transition from each vibration state to only anotheradjacent vibration state in order of magnitude of vibration.

That is, it is possible to reduce a discomfort with the change in theexposure time by changing the exposure time in stages, rather thanabruptly changing the exposure time according to the vibration state.Furthermore, it is possible to reduce the condition that serves as acriterion in each vibration state.

Furthermore, FIG. 27 illustrates an example of changing the vibrationstate in two stages, and it is possible to change each vibration stateto only one-step preceding or subsequent vibration state. In contrast,for example, in a case of changing the vibration state in two or morestages, it is possible to change each vibration state to two-steppreceding or subsequent vibration state or to change each vibrationstate to three-step preceding or subsequent vibration state.Furthermore, it is also possible to change each vibration state indifferent steps.

Furthermore, as the method of determining the vibration state by thevibration state determination unit 32, the above-described Hall data,integral angle (or angle to be corrected), angular velocity, or acombination thereof can be used, similarly to the method described inthe first exposure time control mode. Furthermore, an average value maybe used instead of the peak value within a frame, and determination maybe performed using a predicted value from data of past several frames.Moreover, in determining whether or not the vibration state is withinthe set threshold, the vibration state determination unit 32 can makedetermination for each axis (pitch or yaw) or using the magnitude of avector sum of the axes.

Even in the transition of the vibration state illustrated in FIGS. 25and 27, an image in which no exposure blur occurs in a capture resultcan be obtained on a steady basis by controlling the exposure timeaccording to the determination result of the vibration state.

Note that, in the imaging device 11A illustrated in FIG. 17, theconfiguration example in which the logic unit 22A includes the exposurecontrol unit 31 and the vibration state determination unit 32 has beendescribed. However, the exposure control unit 31 and the vibration statedetermination unit 32 may be separately configured from the logic unit22A. For example, the exposure control unit 31 and the vibration statedetermination unit 32 can be provided in an imaging device control unitoutside the image sensor 13A. Furthermore, the vibration statedetermination unit 32 may be included in the exposure control unit 31.

Note that each processing described with reference to theabove-described flowchart does not necessarily need to bechronologically processed according to the order described as theflowchart, and includes processing executed in parallel or individually(for example, parallel processing or object processing). Furthermore,the program may be processed by one single CPU or may be processed in adistributed manner by a plurality of CPUs.

Furthermore, in the above-described embodiment, the configuration of theimaging device 11 has been described. However, the present invention canbe applied to a camera module provided with at least the image sensor13, the motion sensor 14, the OIS driver 15, the OIS actuator 16, andthe signal processing circuit 17, or various electronic devices in whichthe aforementioned camera module is mounted.

Moreover, the imaging device 11 may not include the logic unit 22 thatapplies the electronic image stabilization for the image output from theimage sensor 13. That is, the function of the logic unit 22 may beprovided in a unit different from the imaging device 11, and image datato which the positional information of the lens unit 12 and the angularvelocity data synchronized with a position on the image are added mayjust be output to the unit. Of course, by adopting the configuration inwhich the imaging device 11 includes the logic unit 22, more favorably,the configuration in which the signal processing circuit 17 is includedin the semiconductor chip of the stacked image sensor 13, the imagestabilization processing can be executed with high accuracy, and thesystem can be easily constructed.

Note that, in the above-described embodiment, the camera shake occurringin the imaging device 11 (that is, the vibration of the image sensor 13built in the imaging device 11) is defined by rotation in the pitchdirection, yaw direction, and roll direction, as illustrated in FIG. 28.

<Use Examples of Image Sensor>

FIG. 29 is a diagram illustrating use examples of the above-describedimage sensor.

The above-described image sensor can be used in various cases of sensinglight such as visible light, infrared light, ultraviolet light, and anX-rays, for example as follows.

-   -   Devices that capture an image provided for appreciation use,        such as a digital camera or a mobile device with a camera        function.    -   Devices provided for transportation, such as in-vehicle sensors        that capture the front, rear, peripheries, an interior of the        vehicle, etc. for safe driving such as automatic stop,        recognition of a state of a driver, or the like, monitoring        cameras that monitor traveling vehicles and roads, distance        measuring sensors that measure a distance between vehicles, and        the like    -   Devices provided for home appliances such as TVs, refrigerators,        and air conditioners to capture gestures of users and perform        device operations according to the gestures    -   Devices provided for medical and healthcare, such as endoscopes,        devices that perform angiography by receiving infrared light    -   Devices provided for security, such as monitoring cameras for        crime prevention and cameras for person authentication use    -   Devices for beauty, such as skin measuring instruments that        capture skin and microscopes that capture scalp    -   Devices provided for sports or the like, such as action cameras        and wearable cameras for sport use    -   Devices provided for agriculture, such as cameras for monitoring        the condition of fields and crops

Note that the present technology can also have the followingconfigurations.

(1)

An imaging device including:

a state determination unit configured to determine a state of a motionof an imaging unit that performs imaging to acquire an image via anoptical system that collects light; and

an exposure control unit configured to perform at least control for anexposure time of the imaging unit according to a determination result bythe state determination unit, in which

relative driving for the optical system or the imaging unit is performedto optically correct a blur appearing in the image according to anexposure period of one frame by the exposure time, and

driving for resetting a relative positional relationship between theoptical system and the imaging unit, the relative positionalrelationship being caused during the exposure period, is performedaccording to a non-exposure period in which exposure is not performedbetween the frames.

(2)

The imaging device according to (1), in which,

in a case where the exposure control unit has performed control formaking the exposure time of the imaging unit shorter than an optimumexposure time in which the image is able to be captured with optimumbrightness, the exposure control unit obtains a gain that compensatesfor a shortage from the optimum brightness and amplifies brightness ofthe image.

(3)

The imaging device according to (2), in which,

in a case where the state determination unit determines that the motionof the imaging unit has a magnitude within a correctable range by theoptical correction,

the exposure control unit obtains the optimum exposure time, using amaximum value of the exposure time of the imaging unit as a maximumexposure time allowed within a frame rate, and sets the obtained optimumexposure time for the imaging unit.

(4)

The imaging device according to (3), in which,

in a case where the state determination unit determines that there is apossibility that the motion of the imaging unit exceeds the correctablerange by the optical correction,

the exposure control unit obtains the exposure time, using a firstthreshold exposure time in which the non-exposure period where therelative positional relationship between the optical system and theimaging unit is resettable is securable as the maximum exposure time,and sets the obtained exposure time for the imaging unit.

(5)

The imaging device according to (4), in which,

in a case where the state determination unit determines that there is apossibility that the motion of the imaging unit exceeds the correctablerange due to optical correction for the image within the exposure periodof one frame even in a case of exposure in the first threshold exposuretime,

the exposure control unit obtains the exposure time, using a secondthreshold exposure time shorter than the first threshold exposure timeas the maximum exposure time, and sets the obtained exposure time forthe imaging unit.

(6)

The imaging device according to any one of (1) to (5), in which

the state determination unit determines the state of the motion of theimaging unit, using positional information of the optical system drivenfor performing the optical correction.

(7)

The imaging device according to any one of (1) to (6), in which

the state determination unit determines the state of the motion of theimaging unit, using an angular velocity detected by a detection unitthat physically detects the motion of the imaging unit or an integralangle of vibration calculated from the angular velocity.

(8)

The imaging device according to any one of (1) to (7), furtherincluding:

a drive control unit configured to obtain a moving amount of whenrelatively moving at least one of the optical system or the imaging unitand optically correcting a blur appearing in an image captured by theimaging unit on the basis of the physically detected motion of theimaging unit, and control driving of at least one of the optical systemor the imaging unit;

a signal processing unit configured to apply signal processing forcorrecting an influence of the motion of the imaging unit on the imageaccording to a function for converting a position on the basis ofpositional information and motion information synchronized for eachcoordinate on the image on the basis of the positional information thatis a detected position of the optical system or the imaging unit drivenunder control by the drive control unit and the motion informationindicating the physically detected motion of the imaging unit;

a logic unit configured to supply, to the signal processing unit, thepositional information and the motion information, and timinginformation indicating timing to synchronize the positional informationand the motion information with the coordinate on the image, togetherwith the image captured by the imaging unit; and

a drive unit configured to drive at least one of the optical system orthe imaging unit according to the moving amount obtained by the drivecontrol unit, detect the position of the optical system or the imagingunit according to the driving, and supply the positional information tothe drive control unit.

(9)

The imaging device according to (8), in which

the logic unit generates control information instructing execution orstop of the optical correction and supplies the generated controlinformation to the drive control unit according to exposure timing toperform exposure by the imaging unit, and

the drive control unit controls, on the basis of the controlinformation,

relative driving for the optical system or the imaging unit to opticallycorrect a blur appearing in the image according to the exposure periodof one frame by the exposure time, and

driving for resetting the relative positional relationship between theoptical system and the imaging unit, the relative positionalrelationship being caused during the exposure period, according to thenon-exposure period.

(10)

A solid-state imaging element configured by stacking:

a semiconductor chip on which an imaging unit that performs imaging toacquire an image via an optical system that collects light is formed;and

a semiconductor chip on which a logic unit including a statedetermination unit that determines a state of a motion of the imagingunit and an exposure control unit that performs at least control for anexposure time of the imaging unit according to a determination result bythe state determination unit is formed, in which

relative driving for the optical system or the imaging unit is performedto optically correct a blur appearing in the image according to anexposure period of one frame by the exposure time, and

driving for resetting a relative positional relationship between theoptical system and the imaging unit, the relative positionalrelationship being caused during the exposure period, is performedaccording to a non-exposure period in which exposure is not performedbetween the frames.

(11)

A camera module including:

an optical system configured to collect light;

an imaging unit configured to perform imaging via the optical system toacquire an image;

a state determination unit configured to determine a state of a motionof the imaging unit; and

an exposure control unit configured to perform at least control for anexposure time of the imaging unit according to a determination result bythe state determination unit, in which

relative driving for the optical system or the imaging unit is performedto optically correct a blur appearing in the image according to anexposure period of one frame by the exposure time, and

driving for resetting a relative positional relationship between theoptical system and the imaging unit, the relative positionalrelationship being caused during the exposure period, is performedaccording to a non-exposure period in which exposure is not performedbetween the frames.

(12)

A drive control unit in which

at least control for an exposure time of an imaging unit that performsimaging to acquire an image via an optical system that collects light isperformed according to a determination result of a state of a motion ofthe imaging unit,

the drive control unit configured to

control relative driving for the optical system or the imaging unit tooptically correct a blur appearing in the image according to an exposureperiod of one frame by the exposure time; and

control driving for resetting a relative positional relationship betweenthe optical system and the imaging unit, the relative positionalrelationship being caused during the exposure period, according to anon-exposure period in which exposure is not performed between theframes.

(13)

An imaging method including:

by an imaging device including an imaging unit that performs imaging toacquire an image via an optical system that collects light,

determining a state of a motion of the imaging unit; and

performing at least control for an exposure time of the imaging unitaccording to a determination result by the state determination unit, inwhich

relative driving for the optical system or the imaging unit is performedto optically correct a blur appearing in the image according to anexposure period of one frame by the exposure time, and

driving for resetting a relative positional relationship between theoptical system and the imaging unit, the relative positionalrelationship being caused during the exposure period, is performedaccording to a non-exposure period in which exposure is not performedbetween the frames.

Note that the present embodiments are not limited to the above-describedembodiments, and various modifications can be made without departingfrom the gist of the present disclosure. Furthermore, the effectsdescribed in the present specification are merely examples and are notlimited, and other effects may be exhibited.

REFERENCE SIGNS LIST

-   11 Imaging device-   12 Lens unit-   13 Image sensor-   14 Motion sensor-   15 OIS driver-   16 OIS actuator-   17 Signal processing circuit-   18 Display-   19 Recoding medium-   21 Imaging unit-   22 Logic unit-   31 Exposure control unit-   32 Vibration state determination unit

The invention claimed is:
 1. An imaging device, comprising: at least onecentral processing unit (CPU) configured to: determine a state of amotion of an imaging circuitry that performs imaging to acquire an imagevia a plurality of lens that collects light; perform at least controlfor an exposure time of the imaging circuitry according to thedetermined state, wherein relative driving for the plurality of lens orthe imaging circuitry is performed to optically correct a blur appearingin the image according to an exposure period of one frame by theexposure time, and driving for resetting a relative positionalrelationship between the plurality of lens and the imaging circuitry,the relative positional relationship being caused during the exposureperiod, is performed according to a non-exposure period in whichexposure is not performed between the frames; obtain, based on themotion of the imaging circuitry has a magnitude within a correctablerange by the optical correction, an optimum exposure time in which theimage is able to be captured with optimum brightness, wherein theoptimum exposure time is obtained based on a maximum value of theexposure time of the imaging circuitry as a maximum exposure timeallowed within a frame rate; and set the obtained optimum exposure timefor the imaging circuitry.
 2. The imaging device according to claim 1,wherein, the at least one CPU is further configured to obtain, based onthe exposure time of the imaging circuitry is shorter than the optimumexposure time, a gain that compensates for a shortage from the optimumbrightness and amplifies brightness of the image.
 3. The imaging deviceaccording to claim 1, wherein the at least one CPU is further configuredto: obtain, based on a possibility that the motion of the imagingcircuitry exceeds the correctable range by the optical correction, theexposure time, using a first threshold exposure time in which thenon-exposure period where the relative positional relationship betweenthe plurality of lens and the imaging circuitry is resettable issecurable as the maximum exposure time; and set the obtained exposuretime for the imaging circuitry.
 4. The imaging device according to claim3, wherein the at least one CPU is further configured to: obtain, basedon a possibility that the motion of the imaging circuitry exceeds thecorrectable range due to the optical correction for the image within theexposure period of one frame even in a case of exposure in the firstthreshold exposure time, the exposure time, using a second thresholdexposure time shorter than the first threshold exposure time as themaximum exposure time; and set the obtained exposure time for theimaging circuitry.
 5. The imaging device according to claim 1, whereinthe at least one CPU is further configured to determine the state of themotion of the imaging circuitry, using positional information of theplurality of lens driven for performing the optical correction.
 6. Theimaging device according to claim 1, wherein the at least one CPU isfurther configured to determine the state of the motion of the imagingcircuitry, using an angular velocity of the imaging circuitry or anintegral angle of vibration calculated from the angular velocity.
 7. Theimaging device according to claim 1, wherein the at least one CPU isfurther configured to: obtain a moving amount of when relatively movingat least one of the plurality of lens or the imaging circuitry;optically correct a blur appearing in the image captured by the imagingcircuitry based on the motion of the imaging circuitry; apply signalprocessing for correcting an influence of the motion of the imagingcircuitry on the image according to a function for converting a positionbased on positional information and motion information synchronized foreach coordinate on the image based on the positional information that isa detected position of the plurality of lens or the imaging circuitryand the motion information; control drive of at least one of theplurality of lens or the imaging circuitry according to the movingamount; and detect the position of the plurality of lens or the imagingcircuitry according to the driving.
 8. The imaging device according toclaim 7, wherein the at least one CPU is configured to: generate controlinformation instructing execution or stop of the optical correction;control, based on the control information, relative driving for theplurality of lens or the imaging circuitry to optically correct a blurappearing in the image according to the exposure period of one frame bythe exposure time; and control driving for resetting the relativepositional relationship between the plurality of lens and the imagingcircuitry, the relative positional relationship being caused during theexposure period, according to the non-exposure period.
 9. A solid-stateimaging element configured by stacking: a semiconductor chip on which animaging circuitry, that performs imaging to acquire an image via aplurality of lens that collects light, is formed; and a semiconductorchip on which at least one central processing unit (CPU) is formed,wherein the at least one CPU is configured to: determine a state of amotion of the imaging circuitry; perform at least control for anexposure time of the imaging circuitry according to the determinedstate, wherein relative driving for the plurality of lens or the imagingcircuitry is performed to optically correct a blur appearing in theimage according to an exposure period of one frame by the exposure time,and driving for resetting a relative positional relationship between theplurality of lens and the imaging circuitry, the relative positionalrelationship being caused during the exposure period, is performedaccording to a non-exposure period in which exposure is not performedbetween the frames; obtain, based on the motion of the imaging circuitryhas a magnitude within a correctable range by the optical correction, anoptimum exposure time in which the image is able to be captured withoptimum brightness, wherein the optimum exposure time is obtained basedon a maximum value of the exposure time of the imaging circuitry as amaximum exposure time allowed within a frame rate; and set the obtainedoptimum exposure time for the imaging circuitry.
 10. A camera module,comprising: a plurality of lens configured to collect light; an imagingcircuitry configured to perform imaging via the plurality of lens toacquire an image; and at least one central processing unit (CPU)configured to: determine a state of a motion of the imaging circuitry;perform at least control for an exposure time of the imaging circuitryaccording to the determined state, wherein relative driving for theplurality of lens or the imaging circuitry is performed to opticallycorrect a blur appearing in the image according to an exposure period ofone frame by the exposure time, and driving for resetting a relativepositional relationship between the plurality of lens and the imagingcircuitry, the relative positional relationship being caused during theexposure period, is performed according to a non-exposure period inwhich exposure is not performed between the frames; obtain, based on themotion of the imaging circuitry has a magnitude within a correctablerange by the optical correction, an optimum exposure time in which theimage is able to be captured with optimum brightness, wherein theoptimum exposure time is obtained based on a maximum value of theexposure time of the imaging circuitry as a maximum exposure timeallowed within a frame rate; and set the obtained optimum exposure timefor the imaging circuitry.
 11. A drive control apparatus in which atleast one central processing unit (CPU) configured to: control anexposure time of an imaging circuitry that performs imaging to acquirean image via a plurality of lens that collects light is performedaccording to a state of a motion of the imaging circuitry; controlrelative driving for the plurality of lens or the imaging circuitry tooptically correct a blur appearing in the image according to an exposureperiod of one frame by the exposure time; control driving for resettinga relative positional relationship between the plurality of lens and theimaging circuitry, the relative positional relationship being causedduring the exposure period, according to a non-exposure period in whichexposure is not performed between the frames; obtain, based on themotion of the imaging circuitry has a magnitude within a correctablerange by the optical correction, an optimum exposure time in which theimage is able to be captured with optimum brightness, wherein theoptimum exposure time is obtained based on a maximum value of theexposure time of the imaging circuitry as a maximum exposure timeallowed within a frame rate; and set the obtained optimum exposure timefor the imaging circuitry.
 12. An imaging method, comprising: by animaging device including an imaging circuitry that performs imaging toacquire an image via a plurality of lens that collects light,determining a state of a motion of the imaging circuitry; performing atleast control for an exposure time of the imaging circuitry according tothe determined state, wherein relative driving for the plurality of lensor the imaging circuitry is performed to optically correct a blurappearing in the image according to an exposure period of one frame bythe exposure time, and driving for resetting a relative positionalrelationship between the plurality of lens and the imaging circuitry,the relative positional relationship being caused during the exposureperiod, is performed according to a non-exposure period in whichexposure is not performed between the frames; obtaining, based on themotion of the imaging circuitry has a magnitude within a correctablerange by the optical correction, an optimum exposure time in which theimage is able to be captured with optimum brightness, wherein theoptimum exposure time is obtained based on a maximum value of theexposure time of the imaging circuitry as a maximum exposure timeallowed within a frame rate; and setting the obtained optimum exposuretime for the imaging circuitry.